[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:23:15 dist-test-slave-lgr5 krb5kdc[29524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:23:15 dist-test-slave-lgr5 krb5kdc[29524](info): set up 2 sockets
Aug 06 12:23:15 dist-test-slave-lgr5 krb5kdc[29524](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 12:23:17.109750 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.008s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:23:17 dist-test-slave-lgr5 krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:23:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:23:17.272641 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34605
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:17.409484 29540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:17.410099 29540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:17.410197 29540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:17.414724 29540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:17.414819 29540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:17.414839 29540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:17.414856 29540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:17.414871 29540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:17.421864 29540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:34605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:17.423779 29540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:17.425575 29540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:17.441812 29554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:17.442557 29549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:17.442655 29548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:17.442881 29540 server_base.cc:1034] running on GCE node
I20260806 12:23:17.443864 29540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:17.446144 29540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:17.447436 29540 hybrid_clock.cc:648] HybridClock initialized: now 1786018997447412 us; error 45 us; skew 500 ppm
Aug 06 12:23:17 dist-test-slave-lgr5 krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:17.451017 29540 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:23:17.452739 29540 webserver.cc:460] Webserver started at http://127.28.210.126:43243/ using document root <none> and password file <none>
I20260806 12:23:17.453827 29540 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:17.453924 29540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:17.454424 29540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:17.457513 29540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "9186fc2c705b42b1bb79e169913a51d3"
format_stamp: "Formatted at 2026-08-06 12:23:17 on dist-test-slave-lgr5"
server_key: "2313ce913346f7afd0e5a89572a1c749"
server_key_iv: "9d1036107b078545bc269eb4e92d6fcd"
server_key_version: "encryptionkey@0"
I20260806 12:23:17.458526 29540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "9186fc2c705b42b1bb79e169913a51d3"
format_stamp: "Formatted at 2026-08-06 12:23:17 on dist-test-slave-lgr5"
server_key: "2313ce913346f7afd0e5a89572a1c749"
server_key_iv: "9d1036107b078545bc269eb4e92d6fcd"
server_key_version: "encryptionkey@0"
I20260806 12:23:17.464679 29540 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260806 12:23:17.468612 29567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:17.470746 29540 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 12:23:17.471031 29540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "9186fc2c705b42b1bb79e169913a51d3"
format_stamp: "Formatted at 2026-08-06 12:23:17 on dist-test-slave-lgr5"
server_key: "2313ce913346f7afd0e5a89572a1c749"
server_key_iv: "9d1036107b078545bc269eb4e92d6fcd"
server_key_version: "encryptionkey@0"
I20260806 12:23:17.471194 29540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:17.490660 29540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:17.495644 29540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:17.495965 29540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:17.512518 29540 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:34605
I20260806 12:23:17.512596 29653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:34605 every 8 connection(s)
I20260806 12:23:17.514448 29540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:17.521196 29655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:17.523962 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 29540
I20260806 12:23:17.524237 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:17.526633 29513 external_mini_cluster.cc:1428] Setting key 0939e4bb196cdd85facf82bf588bed63
I20260806 12:23:17.536237 29655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3: Bootstrap starting.
I20260806 12:23:17.542338 29655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:17.544858 29655 log.cc:824] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:17.548897 29655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3: No bootstrap required, opened a new log
I20260806 12:23:17.556540 29655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9186fc2c705b42b1bb79e169913a51d3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34605 } }
I20260806 12:23:17.556995 29655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:17.557124 29655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9186fc2c705b42b1bb79e169913a51d3, State: Initialized, Role: FOLLOWER
I20260806 12:23:17.558125 29655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9186fc2c705b42b1bb79e169913a51d3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34605 } }
I20260806 12:23:17.558459 29655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:17.558636 29655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:17.558806 29655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 12:23:17 dist-test-slave-lgr5 krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:17.561151 29655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9186fc2c705b42b1bb79e169913a51d3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34605 } }
I20260806 12:23:17.561964 29655 leader_election.cc:302] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9186fc2c705b42b1bb79e169913a51d3; no voters:
I20260806 12:23:17.562621 29655 leader_election.cc:288] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:17.562908 29663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:17.563233 29663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [term 1 LEADER]: Becoming Leader. State: Replica: 9186fc2c705b42b1bb79e169913a51d3, State: Running, Role: LEADER
I20260806 12:23:17.563787 29663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9186fc2c705b42b1bb79e169913a51d3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34605 } }
I20260806 12:23:17.564842 29655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:17.569614 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:17.537076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40726 (local address 127.28.210.126:34605)
0806 12:23:17.538081 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:17.538153 (+ 72us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:17.538269 (+ 116us) server_negotiation.cc:407] Connection header received
0806 12:23:17.543676 (+ 5407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:17.543707 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:17.544361 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:17.544885 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:17.549005 (+ 4120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:17.551017 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:17.553098 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:17.553940 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:17.554239 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:17.560297 (+ 6058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:17.560345 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:17.560363 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:17.560415 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:17.563706 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:17.564712 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:17.564721 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:17.564729 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:17.564960 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:17.565532 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:17.565536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:17.565538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:17.566337 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:17.566571 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:17.567535 (+ 964us) server_negotiation.cc:299] Negotiation successful
0806 12:23:17.567954 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":304,"threads_started":1}
I20260806 12:23:17.571803 29669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9186fc2c705b42b1bb79e169913a51d3. Latest consensus state: current_term: 1 leader_uuid: "9186fc2c705b42b1bb79e169913a51d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9186fc2c705b42b1bb79e169913a51d3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34605 } } }
I20260806 12:23:17.571842 29667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9186fc2c705b42b1bb79e169913a51d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9186fc2c705b42b1bb79e169913a51d3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34605 } } }
I20260806 12:23:17.572018 29669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:17.572027 29667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3 [sys.catalog]: This master's current role is: LEADER
W20260806 12:23:17.575841 29689 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:23:17.576035 29689 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:23:17.576221 29692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:17.578231 29692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:17.588397 29692 catalog_manager.cc:1347] Generated new cluster ID: 9788d75b8c1e41fdaf97429781b98a5c
I20260806 12:23:17.588605 29692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:17.603667 29692 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:17.605342 29692 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:17.613646 29692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9186fc2c705b42b1bb79e169913a51d3: Generated new TSK 0
I20260806 12:23:17.615200 29692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:23:17.668646 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34605
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:17.819026 29704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:17.819449 29704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:17.819501 29704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:17.824203 29704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:17.824306 29704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:17.824369 29704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:23:17.830819 29704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:17.832228 29704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:17.833374 29704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:17.844832 29711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:17.844789 29716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:17.844780 29712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:17.845229 29704 server_base.cc:1034] running on GCE node
I20260806 12:23:17.845652 29704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:17.846339 29704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:17.847527 29704 hybrid_clock.cc:648] HybridClock initialized: now 1786018997847503 us; error 34 us; skew 500 ppm
Aug 06 12:23:17 dist-test-slave-lgr5 krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:17.850715 29704 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:23:17.851900 29704 webserver.cc:460] Webserver started at http://127.28.210.65:45907/ using document root <none> and password file <none>
I20260806 12:23:17.852576 29704 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:17.852648 29704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:17.852857 29704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:17.855247 29704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "cc6050693d81449fad56586c727e1e7c"
format_stamp: "Formatted at 2026-08-06 12:23:17 on dist-test-slave-lgr5"
server_key: "3356564ddd890044f06dc5e649fb790e"
server_key_iv: "2bf39db03b5c06b18e60f68910d3cd58"
server_key_version: "encryptionkey@0"
I20260806 12:23:17.855950 29704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "cc6050693d81449fad56586c727e1e7c"
format_stamp: "Formatted at 2026-08-06 12:23:17 on dist-test-slave-lgr5"
server_key: "3356564ddd890044f06dc5e649fb790e"
server_key_iv: "2bf39db03b5c06b18e60f68910d3cd58"
server_key_version: "encryptionkey@0"
I20260806 12:23:17.860522 29704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 12:23:17.863469 29727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:17.864912 29704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:17.865079 29704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "cc6050693d81449fad56586c727e1e7c"
format_stamp: "Formatted at 2026-08-06 12:23:17 on dist-test-slave-lgr5"
server_key: "3356564ddd890044f06dc5e649fb790e"
server_key_iv: "2bf39db03b5c06b18e60f68910d3cd58"
server_key_version: "encryptionkey@0"
I20260806 12:23:17.865240 29704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:17.882992 29704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:17.887601 29704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:17.887905 29704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:17.889093 29704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:17.890775 29704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:17.890852 29704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:17.890928 29704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:17.890959 29704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:17.907614 29704 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:33465
I20260806 12:23:17.907650 29902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:33465 every 8 connection(s)
I20260806 12:23:17.908733 29704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:23:17.909147 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 29704
I20260806 12:23:17.909271 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:23:17.909596 29513 external_mini_cluster.cc:1428] Setting key 197c7c67f7a32a6eda47efcc63d15324
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
Aug 06 12:23:17 dist-test-slave-lgr5 krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:17.929370 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:17.911458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:36045 (local address 127.28.210.126:34605)
0806 12:23:17.911616 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:17.911621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:17.912730 (+ 1109us) server_negotiation.cc:407] Connection header received
0806 12:23:17.914494 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:17.914498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:17.914564 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:17.914663 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:17.916057 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:17.916906 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:17.918392 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:17.918765 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:17.918885 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:17.925343 (+ 6458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:17.925347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:17.925349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:17.925384 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:17.926875 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:17.927483 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:17.927487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:17.927489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:17.927574 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:17.928069 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:17.928073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:17.928075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:17.928347 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:17.928446 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:17.929038 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 12:23:17.929186 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 12:23:17.930354 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:17.911911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34605 (local address 127.28.210.65:36045)
0806 12:23:17.912543 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:17.912604 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:17.914246 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:17.914824 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:17.914831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:17.915274 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:17.915877 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:17.915895 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:17.917058 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:17.917061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:17.918238 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:17.918250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:17.918963 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:17.918966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:17.919164 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:17.919878 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:17.919896 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:17.925084 (+ 5188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:17.927033 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:17.927041 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:17.927062 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:17.927374 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:17.927667 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:17.927671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:17.927673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:17.927937 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:17.928468 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:17.928476 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:17.928776 (+ 300us) client_negotiation.cc:769] Sending connection context
0806 12:23:17.928978 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 12:23:17.929372 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":190,"threads_started":1}
I20260806 12:23:17.931785 29903 heartbeater.cc:347] Connected to a master server at 127.28.210.126:34605
I20260806 12:23:17.932126 29903 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:17.932809 29903 heartbeater.cc:511] Master 127.28.210.126:34605 requested a full tablet report, sending...
I20260806 12:23:17.935222 29597 ts_manager.cc:194] Registered new tserver with Master: cc6050693d81449fad56586c727e1e7c (127.28.210.65:33465)
I20260806 12:23:17.936851 29597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:36045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:23:17.944185 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34605
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:18.111804 29910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:18.112147 29910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:18.112196 29910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:18.116693 29910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:18.116791 29910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:18.116875 29910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:23:18.123409 29910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:18.124794 29910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:18.126010 29910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:18.137917 29921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:18.137935 29917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:18.137907 29918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:18.138335 29910 server_base.cc:1034] running on GCE node
I20260806 12:23:18.138962 29910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:18.139768 29910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:18.140941 29910 hybrid_clock.cc:648] HybridClock initialized: now 1786018998140895 us; error 60 us; skew 500 ppm
Aug 06 12:23:18 dist-test-slave-lgr5 krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018998, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:18.144210 29910 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:23:18.145224 29910 webserver.cc:460] Webserver started at http://127.28.210.66:41983/ using document root <none> and password file <none>
I20260806 12:23:18.145854 29910 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:18.145929 29910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:18.146162 29910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:18.148799 29910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "4cf2ecc4eb664b19a4a369454dd08a6c"
format_stamp: "Formatted at 2026-08-06 12:23:18 on dist-test-slave-lgr5"
server_key: "51fdb80c3fbf99cf8797a272cd8517c0"
server_key_iv: "f998a1b8056387ffdc184ebecdaddb45"
server_key_version: "encryptionkey@0"
I20260806 12:23:18.149430 29910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "4cf2ecc4eb664b19a4a369454dd08a6c"
format_stamp: "Formatted at 2026-08-06 12:23:18 on dist-test-slave-lgr5"
server_key: "51fdb80c3fbf99cf8797a272cd8517c0"
server_key_iv: "f998a1b8056387ffdc184ebecdaddb45"
server_key_version: "encryptionkey@0"
I20260806 12:23:18.153652 29910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:23:18.156600 29930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:18.158344 29910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 12:23:18.158504 29910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "4cf2ecc4eb664b19a4a369454dd08a6c"
format_stamp: "Formatted at 2026-08-06 12:23:18 on dist-test-slave-lgr5"
server_key: "51fdb80c3fbf99cf8797a272cd8517c0"
server_key_iv: "f998a1b8056387ffdc184ebecdaddb45"
server_key_version: "encryptionkey@0"
I20260806 12:23:18.158641 29910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:18.177956 29910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:18.181867 29910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:18.182145 29910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:18.183054 29910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:18.184157 29910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:18.184216 29910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:18.184275 29910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:18.184302 29910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:18.206403 29910 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:41843
I20260806 12:23:18.206490 30113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:41843 every 8 connection(s)
I20260806 12:23:18.207527 29910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:23:18.215126 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 29910
I20260806 12:23:18.215250 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:23:18.215536 29513 external_mini_cluster.cc:1428] Setting key 7bd792261595b3e5adbd8858e7af3dea
Aug 06 12:23:18 dist-test-slave-lgr5 krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018998, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:18.226377 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:18.210016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:39153 (local address 127.28.210.126:34605)
0806 12:23:18.210173 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:18.210176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:18.211353 (+ 1177us) server_negotiation.cc:407] Connection header received
0806 12:23:18.213739 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:18.213742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:18.213804 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:18.213991 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:18.216054 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:18.216749 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:18.218763 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:18.219089 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:18.219219 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:18.222519 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:18.222523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:18.222525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:18.222556 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:18.224026 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:18.224610 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:18.224613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:18.224614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:18.224753 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:18.225112 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:18.225115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:18.225116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:18.225382 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:18.225479 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:18.226050 (+ 571us) server_negotiation.cc:299] Negotiation successful
0806 12:23:18.226190 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:23:18.227689 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:18.210362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34605 (local address 127.28.210.66:39153)
0806 12:23:18.211130 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:18.211182 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:18.213411 (+ 2229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:18.214199 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:18.214210 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:18.214861 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:18.215797 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:18.215824 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:18.216920 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:18.216924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:18.218577 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:18.218595 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:18.219461 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:18.219467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:18.219779 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:18.220502 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:18.220522 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:18.222310 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:18.224178 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:18.224185 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:18.224207 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:18.224504 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:18.224848 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:18.224851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:18.224853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:18.225016 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:18.225494 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:18.225502 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:18.225791 (+ 289us) client_negotiation.cc:769] Sending connection context
0806 12:23:18.225986 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 12:23:18.226372 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":239,"threads_started":1}
I20260806 12:23:18.229260 30115 heartbeater.cc:347] Connected to a master server at 127.28.210.126:34605
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
I20260806 12:23:18.229532 30115 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:18.230094 30115 heartbeater.cc:511] Master 127.28.210.126:34605 requested a full tablet report, sending...
I20260806 12:23:18.231329 29597 ts_manager.cc:194] Registered new tserver with Master: 4cf2ecc4eb664b19a4a369454dd08a6c (127.28.210.66:41843)
I20260806 12:23:18.232017 29597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:39153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:23:18.258112 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34605
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:18.424016 30121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:18.424466 30121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:18.424520 30121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:18.429029 30121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:18.429126 30121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:18.429196 30121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:23:18.435660 30121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:18.437234 30121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:18.438515 30121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:18.450598 30132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:18.450630 30128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:18.450681 30121 server_base.cc:1034] running on GCE node
W20260806 12:23:18.450625 30138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:18.451543 30121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:18.452397 30121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:18.453608 30121 hybrid_clock.cc:648] HybridClock initialized: now 1786018998453561 us; error 58 us; skew 500 ppm
Aug 06 12:23:18 dist-test-slave-lgr5 krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018998, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:18.457408 30121 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:23:18.458760 30121 webserver.cc:460] Webserver started at http://127.28.210.67:33497/ using document root <none> and password file <none>
I20260806 12:23:18.459484 30121 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:18.459573 30121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:18.459780 30121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:18.462745 30121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "ede6516139be4d44b71e5316a0a10b91"
format_stamp: "Formatted at 2026-08-06 12:23:18 on dist-test-slave-lgr5"
server_key: "a73cdad9c1c6fbe7f282ce93b47478d4"
server_key_iv: "1ce152e648e7a2a5a6afb111a0a514ed"
server_key_version: "encryptionkey@0"
I20260806 12:23:18.463547 30121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "ede6516139be4d44b71e5316a0a10b91"
format_stamp: "Formatted at 2026-08-06 12:23:18 on dist-test-slave-lgr5"
server_key: "a73cdad9c1c6fbe7f282ce93b47478d4"
server_key_iv: "1ce152e648e7a2a5a6afb111a0a514ed"
server_key_version: "encryptionkey@0"
I20260806 12:23:18.468904 30121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 12:23:18.472194 30148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:18.473829 30121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:23:18.473961 30121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "ede6516139be4d44b71e5316a0a10b91"
format_stamp: "Formatted at 2026-08-06 12:23:18 on dist-test-slave-lgr5"
server_key: "a73cdad9c1c6fbe7f282ce93b47478d4"
server_key_iv: "1ce152e648e7a2a5a6afb111a0a514ed"
server_key_version: "encryptionkey@0"
I20260806 12:23:18.474105 30121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:18.489234 30121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:18.493259 30121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:18.493510 30121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:18.494445 30121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:18.495590 30121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:18.495663 30121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:18.495728 30121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:18.495754 30121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:18.518052 30121 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:42825
I20260806 12:23:18.518098 30325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:42825 every 8 connection(s)
I20260806 12:23:18.519433 30121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:23:18.528801 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 30121
I20260806 12:23:18.528977 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:23:18.529304 29513 external_mini_cluster.cc:1428] Setting key 8d16f0f3ebecd1cdd8a8e4b99e5e52fe
Aug 06 12:23:18 dist-test-slave-lgr5 krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018998, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:18.538694 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:18.522070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:53177 (local address 127.28.210.126:34605)
0806 12:23:18.522212 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:18.522217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:18.523684 (+ 1467us) server_negotiation.cc:407] Connection header received
0806 12:23:18.525587 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:18.525591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:18.525659 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:18.525741 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:18.527612 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:18.528219 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:18.530197 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:18.530522 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:18.530632 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:18.534538 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:18.534543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:18.534546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:18.534584 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:18.536147 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:18.536870 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:18.536874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:18.536876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:18.536971 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:18.537359 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:18.537362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:18.537363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:18.537667 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:18.537763 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:18.538393 (+ 630us) server_negotiation.cc:299] Negotiation successful
0806 12:23:18.538524 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 12:23:18.539837 30328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:18.522623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34605 (local address 127.28.210.67:53177)
0806 12:23:18.523449 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:18.523507 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:18.525316 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:18.525967 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:18.525976 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:18.526609 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:18.527425 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:18.527447 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:18.528365 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:18.528368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:18.530050 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:18.530063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:18.530748 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:18.530753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:18.531102 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:18.531979 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:18.532000 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:18.534173 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:18.536318 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:18.536327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:18.536346 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:18.536729 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:18.537056 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:18.537058 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:18.537060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:18.537254 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:18.537792 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:18.537800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:18.538124 (+ 324us) client_negotiation.cc:769] Sending connection context
0806 12:23:18.538342 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 12:23:18.538690 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":261,"threads_started":1}
I20260806 12:23:18.541508 30326 heartbeater.cc:347] Connected to a master server at 127.28.210.126:34605
I20260806 12:23:18.541826 30326 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:18.542441 30326 heartbeater.cc:511] Master 127.28.210.126:34605 requested a full tablet report, sending...
I20260806 12:23:18.543882 29597 ts_manager.cc:194] Registered new tserver with Master: ede6516139be4d44b71e5316a0a10b91 (127.28.210.67:42825)
I20260806 12:23:18.544643 29597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:53177
I20260806 12:23:18.545579 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 12:23:18 dist-test-slave-lgr5 krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 12:23:18 dist-test-slave-lgr5 krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.210.65@KRBTEST.COM
I20260806 12:23:18.582670 30335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:18.571201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.28.210.65:33465)
0806 12:23:18.571582 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:18.571589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:18.571679 (+ 90us) server_negotiation.cc:407] Connection header received
0806 12:23:18.571842 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:18.571848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:18.572070 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:18.572372 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:18.573190 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:18.574056 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:18.574950 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:18.575383 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:18.575480 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:18.578473 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:18.578488 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:18.578505 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:18.578539 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:18.580478 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:18.581028 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:18.581035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:18.581036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:18.581143 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:18.581462 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:18.581467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:18.581469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:18.581789 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:18.582030 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:18.582284 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 12:23:18.582466 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":196,"threads_started":1}
W20260806 12:23:18.583899 29768 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57958
Aug 06 12:23:18 dist-test-slave-lgr5 krb5kdc[29524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 12:23:18 dist-test-slave-lgr5 krb5kdc[29524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786018998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.65@KRBTEST.COM
I20260806 12:23:18.621307 30335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:18.610531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.28.210.65:33465)
0806 12:23:18.610858 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:18.610862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:18.610964 (+ 102us) server_negotiation.cc:407] Connection header received
0806 12:23:18.611139 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:18.611142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:18.611192 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:18.611263 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:18.612200 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:18.612777 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:18.614064 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:18.614340 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:18.614464 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:18.617310 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:18.617314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:18.617316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:18.617354 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:18.618942 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:18.619488 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:18.619492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:18.619494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:18.619596 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:18.620043 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:18.620047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:18.620048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:18.620341 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:18.620475 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:18.620927 (+ 452us) server_negotiation.cc:299] Negotiation successful
0806 12:23:18.621089 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260806 12:23:18.623018 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 29704
I20260806 12:23:18.641718 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 29910
I20260806 12:23:18.674634 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 30121
I20260806 12:23:18.698145 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 29540
2026-08-06T12:23:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2869 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:23:18 dist-test-slave-lgr5 krb5kdc[30348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:23:18 dist-test-slave-lgr5 krb5kdc[30348](info): set up 2 sockets
Aug 06 12:23:18 dist-test-slave-lgr5 krb5kdc[30348](info): commencing operation
krb5kdc: starting...
W20260806 12:23:19.915443 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:23:20 dist-test-slave-lgr5 krb5kdc[30348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:23:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:23:20.073916 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:40957
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:40957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:20.200736 30369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:20.201094 30369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:20.201159 30369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:20.205519 30369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:20.205614 30369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:20.205641 30369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:20.205664 30369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:20.205684 30369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:20.211663 30369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:40957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:40957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:20.213194 30369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:20.214517 30369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:20.224898 30378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:20.225010 30376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:20.224998 30380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:20.225327 30369 server_base.cc:1034] running on GCE node
I20260806 12:23:20.226018 30369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:20.227617 30369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:20.230998 30369 hybrid_clock.cc:648] HybridClock initialized: now 1786019000230958 us; error 53 us; skew 500 ppm
Aug 06 12:23:20 dist-test-slave-lgr5 krb5kdc[30348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:20.234760 30369 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:23:20.236178 30369 webserver.cc:460] Webserver started at http://127.28.210.126:41741/ using document root <none> and password file <none>
I20260806 12:23:20.236864 30369 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:20.236946 30369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:20.237192 30369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:20.239719 30369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "a42f7e7dc9a74eaf923580102be4ea31"
format_stamp: "Formatted at 2026-08-06 12:23:20 on dist-test-slave-lgr5"
server_key: "7013bc0ceb97656057131312cfc7f6d9"
server_key_iv: "4a8819825d10b243eb21b65e9be00807"
server_key_version: "encryptionkey@0"
I20260806 12:23:20.240443 30369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "a42f7e7dc9a74eaf923580102be4ea31"
format_stamp: "Formatted at 2026-08-06 12:23:20 on dist-test-slave-lgr5"
server_key: "7013bc0ceb97656057131312cfc7f6d9"
server_key_iv: "4a8819825d10b243eb21b65e9be00807"
server_key_version: "encryptionkey@0"
I20260806 12:23:20.245277 30369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 12:23:20.248580 30391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:20.250463 30369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 12:23:20.250689 30369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "a42f7e7dc9a74eaf923580102be4ea31"
format_stamp: "Formatted at 2026-08-06 12:23:20 on dist-test-slave-lgr5"
server_key: "7013bc0ceb97656057131312cfc7f6d9"
server_key_iv: "4a8819825d10b243eb21b65e9be00807"
server_key_version: "encryptionkey@0"
I20260806 12:23:20.250854 30369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:20.268356 30369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:20.272150 30369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:20.272367 30369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:20.281466 30369 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:40957
I20260806 12:23:20.281636 30468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:40957 every 8 connection(s)
I20260806 12:23:20.282706 30369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:20.282985 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 30369
I20260806 12:23:20.283118 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:20.283464 29513 external_mini_cluster.cc:1428] Setting key 5a399626c1bd4f4a7d393938e5eddcf3
I20260806 12:23:20.286610 30469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 12:23:20 dist-test-slave-lgr5 krb5kdc[30348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:20.295008 30469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31: Bootstrap starting.
I20260806 12:23:20.298573 30469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:20.299649 30469 log.cc:824] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:20.300580 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:20.285113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.28.210.126:40957)
0806 12:23:20.285607 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:20.285631 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:20.285871 (+ 240us) server_negotiation.cc:407] Connection header received
0806 12:23:20.286775 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:20.286801 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:20.287248 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:20.287655 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:20.288748 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:20.289885 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:20.290860 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:20.291419 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:20.291601 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:20.295133 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:20.295139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:20.295156 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:20.295195 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:20.297160 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:20.297773 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:20.297778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:20.297784 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:20.297907 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:20.298254 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:20.298257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:20.298258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:20.298740 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:20.298893 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:20.299262 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 12:23:20.299569 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":133,"threads_started":1}
I20260806 12:23:20.302069 30469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31: No bootstrap required, opened a new log
I20260806 12:23:20.306699 30469 raft_consensus.cc:348] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42f7e7dc9a74eaf923580102be4ea31" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40957 } }
I20260806 12:23:20.307224 30469 raft_consensus.cc:374] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:20.307389 30469 raft_consensus.cc:729] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42f7e7dc9a74eaf923580102be4ea31, State: Initialized, Role: FOLLOWER
I20260806 12:23:20.308005 30469 consensus_queue.cc:260] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42f7e7dc9a74eaf923580102be4ea31" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40957 } }
I20260806 12:23:20.308240 30469 raft_consensus.cc:388] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:20.308300 30469 raft_consensus.cc:482] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:20.308385 30469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:20.309506 30469 raft_consensus.cc:504] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42f7e7dc9a74eaf923580102be4ea31" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40957 } }
I20260806 12:23:20.309885 30469 leader_election.cc:302] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a42f7e7dc9a74eaf923580102be4ea31; no voters:
I20260806 12:23:20.310362 30469 leader_election.cc:288] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:20.310518 30477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:20.310801 30477 raft_consensus.cc:686] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [term 1 LEADER]: Becoming Leader. State: Replica: a42f7e7dc9a74eaf923580102be4ea31, State: Running, Role: LEADER
I20260806 12:23:20.311151 30477 consensus_queue.cc:237] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42f7e7dc9a74eaf923580102be4ea31" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40957 } }
I20260806 12:23:20.311793 30469 sys_catalog.cc:564] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:20.312944 30479 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a42f7e7dc9a74eaf923580102be4ea31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42f7e7dc9a74eaf923580102be4ea31" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40957 } } }
I20260806 12:23:20.313112 30479 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:20.313453 30481 sys_catalog.cc:455] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a42f7e7dc9a74eaf923580102be4ea31. Latest consensus state: current_term: 1 leader_uuid: "a42f7e7dc9a74eaf923580102be4ea31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a42f7e7dc9a74eaf923580102be4ea31" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40957 } } }
I20260806 12:23:20.313565 30481 sys_catalog.cc:458] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:20.313520 30492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:20.317034 30492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:20.323637 30492 catalog_manager.cc:1347] Generated new cluster ID: 5573e213754c4492ac09b05595dc102f
I20260806 12:23:20.323750 30492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:20.330857 30492 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:20.332120 30492 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:20.350999 30492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a42f7e7dc9a74eaf923580102be4ea31: Generated new TSK 0
I20260806 12:23:20.351843 30492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:23:20.392401 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:40957
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:20.527777 30510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:20.528154 30510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:20.528209 30510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:20.532652 30510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:20.532737 30510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:20.532801 30510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:23:20.538607 30510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:40957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:20.539872 30510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:20.541082 30510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:20.551712 30517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:20.551940 30518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:20.551962 30524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:20.552013 30510 server_base.cc:1034] running on GCE node
I20260806 12:23:20.552583 30510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:20.553294 30510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:20.554467 30510 hybrid_clock.cc:648] HybridClock initialized: now 1786019000554453 us; error 30 us; skew 500 ppm
Aug 06 12:23:20 dist-test-slave-lgr5 krb5kdc[30348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:20.560426 30510 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:23:20.561487 30510 webserver.cc:460] Webserver started at http://127.28.210.65:40525/ using document root <none> and password file <none>
I20260806 12:23:20.562132 30510 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:20.562206 30510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:20.562417 30510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:20.565110 30510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "7576ca6f412545248f9546d5ecf018b1"
format_stamp: "Formatted at 2026-08-06 12:23:20 on dist-test-slave-lgr5"
server_key: "9ab517963bdeeea88b3ef1fe4f4b55f6"
server_key_iv: "052ed0e8b223ee4f83d25b1aba1787ed"
server_key_version: "encryptionkey@0"
I20260806 12:23:20.565820 30510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "7576ca6f412545248f9546d5ecf018b1"
format_stamp: "Formatted at 2026-08-06 12:23:20 on dist-test-slave-lgr5"
server_key: "9ab517963bdeeea88b3ef1fe4f4b55f6"
server_key_iv: "052ed0e8b223ee4f83d25b1aba1787ed"
server_key_version: "encryptionkey@0"
I20260806 12:23:20.570375 30510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:23:20.573146 30533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:20.574612 30510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:20.574743 30510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "7576ca6f412545248f9546d5ecf018b1"
format_stamp: "Formatted at 2026-08-06 12:23:20 on dist-test-slave-lgr5"
server_key: "9ab517963bdeeea88b3ef1fe4f4b55f6"
server_key_iv: "052ed0e8b223ee4f83d25b1aba1787ed"
server_key_version: "encryptionkey@0"
I20260806 12:23:20.574874 30510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:20.591101 30510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:20.594769 30510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:20.594998 30510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:20.595736 30510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:20.596761 30510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:20.596822 30510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:20.596868 30510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:20.596907 30510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:20.611268 30510 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:34227
I20260806 12:23:20.611806 30713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:34227 every 8 connection(s)
I20260806 12:23:20.612735 30510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:23:20.622246 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 30510
I20260806 12:23:20.622368 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:23:20.622628 29513 external_mini_cluster.cc:1428] Setting key b09f3dbc11f4c482a114dbd465617fdc
Aug 06 12:23:20 dist-test-slave-lgr5 krb5kdc[30348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:20.630220 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:20.615676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:52625 (local address 127.28.210.126:40957)
0806 12:23:20.615890 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:20.615894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:20.617147 (+ 1253us) server_negotiation.cc:407] Connection header received
0806 12:23:20.618661 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:20.618665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:20.618752 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:20.618865 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:20.620434 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:20.621005 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:20.622390 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:20.622692 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:20.622810 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:20.626298 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:20.626302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:20.626304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:20.626335 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:20.627783 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:20.628461 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:20.628464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:20.628465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:20.628563 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:20.628938 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:20.628941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:20.628942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:20.629160 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:20.629250 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:20.629864 (+ 614us) server_negotiation.cc:299] Negotiation successful
0806 12:23:20.630037 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 12:23:20.631178 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:20.616160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40957 (local address 127.28.210.65:52625)
0806 12:23:20.616921 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:20.616980 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:20.618357 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:20.619097 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:20.619107 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:20.619556 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:20.620237 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:20.620259 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:20.621164 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:20.621167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:20.622241 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:20.622254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:20.622903 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:20.622907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:20.623169 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:20.624069 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:20.624117 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:20.626065 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:20.627929 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:20.627936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:20.627959 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:20.628337 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:20.628654 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:20.628657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:20.628660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:20.628830 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:20.629272 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:20.629280 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:20.629596 (+ 316us) client_negotiation.cc:769] Sending connection context
0806 12:23:20.629785 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 12:23:20.630192 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":283,"threads_started":1}
I20260806 12:23:20.632750 30715 heartbeater.cc:347] Connected to a master server at 127.28.210.126:40957
I20260806 12:23:20.633109 30715 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:23:20.633996 30715 heartbeater.cc:511] Master 127.28.210.126:40957 requested a full tablet report, sending...
I20260806 12:23:20.636554 30412 ts_manager.cc:194] Registered new tserver with Master: 7576ca6f412545248f9546d5ecf018b1 (127.28.210.65:34227)
I20260806 12:23:20.638175 30412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:52625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:23:20.661100 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:40957
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:20.794787 30723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:20.795131 30723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:20.795181 30723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:20.799870 30723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:20.799974 30723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:20.800071 30723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:23:20.805989 30723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:40957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:20.807232 30723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:20.808394 30723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:20.820752 30735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:20.820793 30723 server_base.cc:1034] running on GCE node
W20260806 12:23:20.820817 30732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:20.820753 30729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:20.821587 30723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:20.822541 30723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:20.823791 30723 hybrid_clock.cc:648] HybridClock initialized: now 1786019000823752 us; error 54 us; skew 500 ppm
Aug 06 12:23:20 dist-test-slave-lgr5 krb5kdc[30348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:20.827579 30723 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:23:20.828931 30723 webserver.cc:460] Webserver started at http://127.28.210.66:43681/ using document root <none> and password file <none>
I20260806 12:23:20.829722 30723 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:20.829814 30723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:20.830016 30723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:20.832583 30723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "71a7b20728f44d4d8ada788548ab74ea"
format_stamp: "Formatted at 2026-08-06 12:23:20 on dist-test-slave-lgr5"
server_key: "1f9d615bb2d808e129be972bb55bc29b"
server_key_iv: "ca49d0e4c2490d98538fcac54381ab70"
server_key_version: "encryptionkey@0"
I20260806 12:23:20.833299 30723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "71a7b20728f44d4d8ada788548ab74ea"
format_stamp: "Formatted at 2026-08-06 12:23:20 on dist-test-slave-lgr5"
server_key: "1f9d615bb2d808e129be972bb55bc29b"
server_key_iv: "ca49d0e4c2490d98538fcac54381ab70"
server_key_version: "encryptionkey@0"
I20260806 12:23:20.837879 30723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:23:20.841032 30746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:20.842561 30723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:23:20.842708 30723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "71a7b20728f44d4d8ada788548ab74ea"
format_stamp: "Formatted at 2026-08-06 12:23:20 on dist-test-slave-lgr5"
server_key: "1f9d615bb2d808e129be972bb55bc29b"
server_key_iv: "ca49d0e4c2490d98538fcac54381ab70"
server_key_version: "encryptionkey@0"
I20260806 12:23:20.842854 30723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:20.879444 30723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:20.884122 30723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:20.884423 30723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:20.885658 30723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:20.887058 30723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:20.887156 30723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:20.887214 30723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:20.887229 30723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:20.899803 30723 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:33225
I20260806 12:23:20.899848 30908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:33225 every 8 connection(s)
I20260806 12:23:20.900882 30723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
Aug 06 12:23:20 dist-test-slave-lgr5 krb5kdc[30348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:20.910977 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 30723
I20260806 12:23:20.911090 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:23:20.911356 29513 external_mini_cluster.cc:1428] Setting key 35b74b7198f222cb0394bd019f71e8b1
I20260806 12:23:20.916010 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:20.903085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:54819 (local address 127.28.210.126:40957)
0806 12:23:20.903228 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:20.903232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:20.904260 (+ 1028us) server_negotiation.cc:407] Connection header received
0806 12:23:20.905258 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:20.905261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:20.905329 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:20.905430 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:20.906798 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:20.907346 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:20.908660 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:20.908891 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:20.908989 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:20.911808 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:20.911813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:20.911816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:20.911850 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:20.913728 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:20.914317 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:20.914321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:20.914323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:20.914411 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:20.914768 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:20.914771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:20.914773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:20.914987 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:20.915090 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:20.915676 (+ 586us) server_negotiation.cc:299] Negotiation successful
0806 12:23:20.915823 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 12:23:20.917001 30913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:20.903499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40957 (local address 127.28.210.66:54819)
0806 12:23:20.904049 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:20.904138 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:20.905031 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:20.905649 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:20.905657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:20.906078 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:20.906624 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:20.906640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:20.907494 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:20.907497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:20.908522 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:20.908535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:20.909060 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:20.909065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:20.909249 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:20.909874 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:20.909894 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:20.911565 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:20.913873 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:20.913880 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:20.913897 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:20.914211 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:20.914507 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:20.914511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:20.914512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:20.914668 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:20.915106 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:20.915114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:20.915406 (+ 292us) client_negotiation.cc:769] Sending connection context
0806 12:23:20.915610 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 12:23:20.916015 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":146,"threads_started":1}
I20260806 12:23:20.918288 30911 heartbeater.cc:347] Connected to a master server at 127.28.210.126:40957
I20260806 12:23:20.918557 30911 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:20.919152 30911 heartbeater.cc:511] Master 127.28.210.126:40957 requested a full tablet report, sending...
I20260806 12:23:20.920425 30412 ts_manager.cc:194] Registered new tserver with Master: 71a7b20728f44d4d8ada788548ab74ea (127.28.210.66:33225)
I20260806 12:23:20.921322 30412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:54819
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:23:20.949178 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:40957
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:21.077606 30917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:21.077975 30917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:21.078050 30917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:21.082775 30917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:21.082894 30917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:21.082994 30917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:23:21.089006 30917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:40957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:21.090245 30917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:21.091459 30917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:21.102597 30925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:21.102594 30930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:21.103073 30924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:21.103205 30917 server_base.cc:1034] running on GCE node
I20260806 12:23:21.103624 30917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:21.104480 30917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:21.105669 30917 hybrid_clock.cc:648] HybridClock initialized: now 1786019001105646 us; error 36 us; skew 500 ppm
Aug 06 12:23:21 dist-test-slave-lgr5 krb5kdc[30348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019001, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:21.108853 30917 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:23:21.109869 30917 webserver.cc:460] Webserver started at http://127.28.210.67:41339/ using document root <none> and password file <none>
I20260806 12:23:21.110491 30917 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:21.110559 30917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:21.110812 30917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:21.113348 30917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "83d9f74eb829414f8a534846027efe47"
format_stamp: "Formatted at 2026-08-06 12:23:21 on dist-test-slave-lgr5"
server_key: "49b53a8ecd0063a42b0dc80d54a6f5c9"
server_key_iv: "2384ed1556d9c23ebe4ecebcca652ae5"
server_key_version: "encryptionkey@0"
I20260806 12:23:21.113991 30917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "83d9f74eb829414f8a534846027efe47"
format_stamp: "Formatted at 2026-08-06 12:23:21 on dist-test-slave-lgr5"
server_key: "49b53a8ecd0063a42b0dc80d54a6f5c9"
server_key_iv: "2384ed1556d9c23ebe4ecebcca652ae5"
server_key_version: "encryptionkey@0"
I20260806 12:23:21.118803 30917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 12:23:21.121795 30940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:21.123359 30917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 12:23:21.123538 30917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "83d9f74eb829414f8a534846027efe47"
format_stamp: "Formatted at 2026-08-06 12:23:21 on dist-test-slave-lgr5"
server_key: "49b53a8ecd0063a42b0dc80d54a6f5c9"
server_key_iv: "2384ed1556d9c23ebe4ecebcca652ae5"
server_key_version: "encryptionkey@0"
I20260806 12:23:21.123711 30917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:21.153893 30917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:21.157773 30917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:21.157990 30917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:21.158671 30917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:21.159636 30917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:21.159688 30917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:21.159731 30917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:21.159745 30917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:21.174515 30917 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:34407
I20260806 12:23:21.174535 31111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:34407 every 8 connection(s)
I20260806 12:23:21.175588 30917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:23:21.179306 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 30917
I20260806 12:23:21.179430 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:23:21.179721 29513 external_mini_cluster.cc:1428] Setting key 639f10a4e72a498e0127e2277e8cdfe3
Aug 06 12:23:21 dist-test-slave-lgr5 krb5kdc[30348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019001, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:21.192303 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.177904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:54253 (local address 127.28.210.126:40957)
0806 12:23:21.178060 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:21.178064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:21.179176 (+ 1112us) server_negotiation.cc:407] Connection header received
0806 12:23:21.180844 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:21.180847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:21.180901 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:21.180971 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:21.182731 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.183314 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.184723 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.184958 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.185038 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.188036 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:21.188040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:21.188042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:21.188074 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:21.189757 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:21.190390 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:21.190395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:21.190396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:21.190500 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:21.190922 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:21.190924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:21.190926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:21.191159 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:21.191244 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:21.191852 (+ 608us) server_negotiation.cc:299] Negotiation successful
0806 12:23:21.192043 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 12:23:21.193645 31114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.178311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40957 (local address 127.28.210.67:54253)
0806 12:23:21.178930 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:21.178984 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:21.180604 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:21.181237 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:21.181246 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:21.181802 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:21.182544 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.182565 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.183567 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.183570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.184583 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.184596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.185100 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.185103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.185259 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.185932 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:21.185951 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:21.187806 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:21.189900 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:21.189908 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:21.189924 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:21.190240 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:21.190609 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:21.190612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:21.190613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:21.190837 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:21.191240 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:21.191247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:21.191591 (+ 344us) client_negotiation.cc:769] Sending connection context
0806 12:23:21.191816 (+ 225us) client_negotiation.cc:240] Negotiation successful
0806 12:23:21.192318 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":147,"threads_started":1}
I20260806 12:23:21.197077 31112 heartbeater.cc:347] Connected to a master server at 127.28.210.126:40957
I20260806 12:23:21.197644 31112 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:21.198491 31112 heartbeater.cc:511] Master 127.28.210.126:40957 requested a full tablet report, sending...
I20260806 12:23:21.200287 30412 ts_manager.cc:194] Registered new tserver with Master: 83d9f74eb829414f8a534846027efe47 (127.28.210.67:34407)
I20260806 12:23:21.201009 30412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:54253
I20260806 12:23:21.207121 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:23:21.220281 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.212052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.28.210.126:40957)
0806 12:23:21.212232 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:21.212236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:21.212596 (+ 360us) server_negotiation.cc:407] Connection header received
0806 12:23:21.212729 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:21.212732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:21.212797 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:21.212884 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:21.213664 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.214198 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.215039 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.215258 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.215350 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.216886 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:21.216889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:21.216891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:21.216918 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:21.218516 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:21.218969 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:21.218972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:21.218974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:21.219057 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:21.219320 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:21.219323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:21.219325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:21.219581 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:21.219708 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:21.219955 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 12:23:21.220114 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 12:23:21.226984 30412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:23:21.230926 30412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:21.249171 31138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.243438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59294 (local address 127.28.210.66:33225)
0806 12:23:21.244361 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:21.244372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:21.244394 (+ 22us) server_negotiation.cc:407] Connection header received
0806 12:23:21.244469 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:21.244474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:21.244646 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:21.245167 (+ 521us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:21.245357 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.246464 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.247789 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.248419 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.248523 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.248752 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:21.248883 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 12:23:21.248998 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":228,"threads_started":1}
I20260806 12:23:21.249176 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.242953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:33225 (local address 127.0.0.1:59294)
0806 12:23:21.243882 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:21.243894 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:21.244114 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:21.244846 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:21.244849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:21.244877 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:21.245023 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.245032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.246588 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.246591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.247671 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.247681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.248547 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.248551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.248667 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.248713 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 12:23:21.248832 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 12:23:21.249012 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":82,"spinlock_wait_cycles":1920,"thread_start_us":208,"threads_started":1}
I20260806 12:23:21.249233 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.243949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.28.210.65:34227)
0806 12:23:21.244391 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:21.244396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:21.244415 (+ 19us) server_negotiation.cc:407] Connection header received
0806 12:23:21.244487 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:21.244492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:21.244646 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:21.245009 (+ 363us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:21.245159 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.246330 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.247726 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.248571 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.248768 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.248943 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:21.248996 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 12:23:21.249090 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":150,"threads_started":1}
I20260806 12:23:21.249773 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.242581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:34407 (local address 127.0.0.1:35594)
0806 12:23:21.242986 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:21.243049 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:21.243305 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:21.244459 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:21.244466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:21.244488 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:21.244648 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.244657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.245987 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.245991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.247468 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.247481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.248666 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.248670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.249449 (+ 779us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.249482 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 12:23:21.249552 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 12:23:21.249637 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":94,"thread_start_us":88,"threads_started":1}
I20260806 12:23:21.249185 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.242813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:34227 (local address 127.0.0.1:51852)
0806 12:23:21.243476 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:21.243489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:21.243704 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:21.244855 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:21.244857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:21.244883 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:21.245046 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.245057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.246464 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.246468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.247602 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.247610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.248685 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.248688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.248787 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.248809 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 12:23:21.248872 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 12:23:21.249005 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":73,"spinlock_wait_cycles":3072,"thread_start_us":337,"threads_started":1}
I20260806 12:23:21.250052 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.243372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.28.210.67:34407)
0806 12:23:21.243757 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:21.243770 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:21.243794 (+ 24us) server_negotiation.cc:407] Connection header received
0806 12:23:21.243880 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:21.243905 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:21.244158 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:21.244811 (+ 653us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:21.244962 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.245769 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.247613 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.248513 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.249522 (+ 1009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.249695 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:21.249751 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 12:23:21.249868 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":143,"threads_started":1}
I20260806 12:23:21.253350 31000 tablet_service.cc:1467] Processing CreateTablet for tablet ecb8719f20614415b0382b83cbeb8d20 (DEFAULT_TABLE table=test-table [id=00d892d0d4004f579fe0e675fa204aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:21.254755 31000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb8719f20614415b0382b83cbeb8d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:21.255172 30806 tablet_service.cc:1467] Processing CreateTablet for tablet ecb8719f20614415b0382b83cbeb8d20 (DEFAULT_TABLE table=test-table [id=00d892d0d4004f579fe0e675fa204aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:21.255661 30605 tablet_service.cc:1467] Processing CreateTablet for tablet ecb8719f20614415b0382b83cbeb8d20 (DEFAULT_TABLE table=test-table [id=00d892d0d4004f579fe0e675fa204aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:21.256572 30806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb8719f20614415b0382b83cbeb8d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:21.256826 30605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb8719f20614415b0382b83cbeb8d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:21.264258 31140 tablet_bootstrap.cc:492] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47: Bootstrap starting.
I20260806 12:23:21.265144 31141 tablet_bootstrap.cc:492] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea: Bootstrap starting.
I20260806 12:23:21.266131 31145 tablet_bootstrap.cc:492] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1: Bootstrap starting.
I20260806 12:23:21.267422 31140 tablet_bootstrap.cc:654] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:21.267639 31141 tablet_bootstrap.cc:654] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:21.269304 31141 log.cc:824] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:21.269528 31140 log.cc:824] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:21.269569 31145 tablet_bootstrap.cc:654] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:21.270924 31145 log.cc:824] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:21.272145 31141 tablet_bootstrap.cc:492] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea: No bootstrap required, opened a new log
I20260806 12:23:21.272169 31140 tablet_bootstrap.cc:492] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47: No bootstrap required, opened a new log
I20260806 12:23:21.272409 31141 ts_tablet_manager.cc:1397] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260806 12:23:21.272409 31140 ts_tablet_manager.cc:1397] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260806 12:23:21.273227 31145 tablet_bootstrap.cc:492] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1: No bootstrap required, opened a new log
I20260806 12:23:21.273463 31145 ts_tablet_manager.cc:1397] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 12:23:21.278779 31140 raft_consensus.cc:348] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d9f74eb829414f8a534846027efe47" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34407 } } peers { permanent_uuid: "7576ca6f412545248f9546d5ecf018b1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34227 } } peers { permanent_uuid: "71a7b20728f44d4d8ada788548ab74ea" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 33225 } }
I20260806 12:23:21.278779 31141 raft_consensus.cc:348] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d9f74eb829414f8a534846027efe47" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34407 } } peers { permanent_uuid: "7576ca6f412545248f9546d5ecf018b1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34227 } } peers { permanent_uuid: "71a7b20728f44d4d8ada788548ab74ea" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 33225 } }
I20260806 12:23:21.279233 31141 raft_consensus.cc:374] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:21.279234 31140 raft_consensus.cc:374] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:21.279309 31140 raft_consensus.cc:729] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d9f74eb829414f8a534846027efe47, State: Initialized, Role: FOLLOWER
I20260806 12:23:21.279309 31141 raft_consensus.cc:729] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a7b20728f44d4d8ada788548ab74ea, State: Initialized, Role: FOLLOWER
I20260806 12:23:21.279181 31145 raft_consensus.cc:348] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d9f74eb829414f8a534846027efe47" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34407 } } peers { permanent_uuid: "7576ca6f412545248f9546d5ecf018b1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34227 } } peers { permanent_uuid: "71a7b20728f44d4d8ada788548ab74ea" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 33225 } }
I20260806 12:23:21.279513 31145 raft_consensus.cc:374] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:21.279580 31145 raft_consensus.cc:729] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7576ca6f412545248f9546d5ecf018b1, State: Initialized, Role: FOLLOWER
I20260806 12:23:21.279906 31140 consensus_queue.cc:260] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d9f74eb829414f8a534846027efe47" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34407 } } peers { permanent_uuid: "7576ca6f412545248f9546d5ecf018b1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34227 } } peers { permanent_uuid: "71a7b20728f44d4d8ada788548ab74ea" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 33225 } }
I20260806 12:23:21.279906 31141 consensus_queue.cc:260] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d9f74eb829414f8a534846027efe47" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34407 } } peers { permanent_uuid: "7576ca6f412545248f9546d5ecf018b1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34227 } } peers { permanent_uuid: "71a7b20728f44d4d8ada788548ab74ea" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 33225 } }
I20260806 12:23:21.280097 31145 consensus_queue.cc:260] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d9f74eb829414f8a534846027efe47" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34407 } } peers { permanent_uuid: "7576ca6f412545248f9546d5ecf018b1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34227 } } peers { permanent_uuid: "71a7b20728f44d4d8ada788548ab74ea" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 33225 } }
I20260806 12:23:21.281286 30911 heartbeater.cc:503] Master 127.28.210.126:40957 was elected leader, sending a full tablet report...
I20260806 12:23:21.281350 31141 ts_tablet_manager.cc:1428] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260806 12:23:21.281428 31140 ts_tablet_manager.cc:1428] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260806 12:23:21.281483 31145 ts_tablet_manager.cc:1428] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20260806 12:23:21.281876 30715 heartbeater.cc:503] Master 127.28.210.126:40957 was elected leader, sending a full tablet report...
I20260806 12:23:21.286388 31112 heartbeater.cc:503] Master 127.28.210.126:40957 was elected leader, sending a full tablet report...
W20260806 12:23:21.364928 30716 tablet.cc:2369] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:23:21.402262 30912 tablet.cc:2369] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:23:21.427009 31113 tablet.cc:2369] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:23:21.491853 31151 raft_consensus.cc:482] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:21.492151 31151 raft_consensus.cc:504] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d9f74eb829414f8a534846027efe47" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34407 } } peers { permanent_uuid: "7576ca6f412545248f9546d5ecf018b1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34227 } } peers { permanent_uuid: "71a7b20728f44d4d8ada788548ab74ea" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 33225 } }
I20260806 12:23:21.493618 31151 leader_election.cc:288] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83d9f74eb829414f8a534846027efe47 (127.28.210.67:34407), 7576ca6f412545248f9546d5ecf018b1 (127.28.210.65:34227)
I20260806 12:23:21.498000 31152 raft_consensus.cc:482] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:21.498351 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.494350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:47087 (local address 127.28.210.65:34227)
0806 12:23:21.494524 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:21.494528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:21.494542 (+ 14us) server_negotiation.cc:407] Connection header received
0806 12:23:21.494603 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:21.494605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:21.494648 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:21.494708 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:21.495114 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.495672 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.497212 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.497778 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.497904 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.497963 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:21.498143 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 12:23:21.498217 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 12:23:21.498370 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.493910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:34227 (local address 127.28.210.66:47087)
0806 12:23:21.494229 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:21.494249 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:21.494437 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:21.494758 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:21.494762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:21.494791 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:21.494961 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.494968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.495801 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.495805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.497047 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.497062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.497904 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.497908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.498021 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.498042 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 12:23:21.498111 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 12:23:21.498193 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"spinlock_wait_cycles":23296,"thread_start_us":124,"threads_started":1}
I20260806 12:23:21.498289 31152 raft_consensus.cc:504] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d9f74eb829414f8a534846027efe47" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34407 } } peers { permanent_uuid: "7576ca6f412545248f9546d5ecf018b1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34227 } } peers { permanent_uuid: "71a7b20728f44d4d8ada788548ab74ea" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 33225 } }
I20260806 12:23:21.499061 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.494149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:34073 (local address 127.28.210.67:34407)
0806 12:23:21.494294 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:21.494299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:21.494468 (+ 169us) server_negotiation.cc:407] Connection header received
0806 12:23:21.494632 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:21.494635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:21.494693 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:21.494789 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:21.495291 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.495872 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.497511 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.498141 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.498301 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.498366 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:21.498737 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 12:23:21.498927 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:23:21.499652 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.493962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:34407 (local address 127.28.210.66:34073)
0806 12:23:21.494291 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:21.494305 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:21.494501 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:21.494812 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:21.494815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:21.494854 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:21.495119 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.495128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.496009 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.496011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.497369 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.497381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.498444 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.498449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.498567 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.498596 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 12:23:21.499391 (+ 795us) client_negotiation.cc:240] Negotiation successful
0806 12:23:21.499495 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"spinlock_wait_cycles":1408,"thread_start_us":96,"threads_started":1}
I20260806 12:23:21.499572 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb8719f20614415b0382b83cbeb8d20" candidate_uuid: "71a7b20728f44d4d8ada788548ab74ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7576ca6f412545248f9546d5ecf018b1" is_pre_election: true
I20260806 12:23:21.500020 31152 leader_election.cc:288] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83d9f74eb829414f8a534846027efe47 (127.28.210.67:34407), 71a7b20728f44d4d8ada788548ab74ea (127.28.210.66:33225)
I20260806 12:23:21.500041 30637 raft_consensus.cc:2454] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a7b20728f44d4d8ada788548ab74ea in term 0.
I20260806 12:23:21.501070 31040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb8719f20614415b0382b83cbeb8d20" candidate_uuid: "71a7b20728f44d4d8ada788548ab74ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d9f74eb829414f8a534846027efe47" is_pre_election: true
I20260806 12:23:21.501487 31040 raft_consensus.cc:2454] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71a7b20728f44d4d8ada788548ab74ea in term 0.
I20260806 12:23:21.501571 30747 leader_election.cc:302] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a7b20728f44d4d8ada788548ab74ea, 7576ca6f412545248f9546d5ecf018b1; no voters:
I20260806 12:23:21.501940 31151 raft_consensus.cc:2781] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:23:21.502043 31151 raft_consensus.cc:482] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:21.502099 31151 raft_consensus.cc:3037] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:21.504024 31151 raft_consensus.cc:504] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d9f74eb829414f8a534846027efe47" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34407 } } peers { permanent_uuid: "7576ca6f412545248f9546d5ecf018b1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34227 } } peers { permanent_uuid: "71a7b20728f44d4d8ada788548ab74ea" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 33225 } }
I20260806 12:23:21.504956 31151 leader_election.cc:288] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [CANDIDATE]: Term 1 election: Requested vote from peers 83d9f74eb829414f8a534846027efe47 (127.28.210.67:34407), 7576ca6f412545248f9546d5ecf018b1 (127.28.210.65:34227)
I20260806 12:23:21.505077 31040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb8719f20614415b0382b83cbeb8d20" candidate_uuid: "71a7b20728f44d4d8ada788548ab74ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d9f74eb829414f8a534846027efe47"
I20260806 12:23:21.505312 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb8719f20614415b0382b83cbeb8d20" candidate_uuid: "71a7b20728f44d4d8ada788548ab74ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7576ca6f412545248f9546d5ecf018b1"
I20260806 12:23:21.505295 31040 raft_consensus.cc:3037] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:21.505463 30637 raft_consensus.cc:3037] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:21.506088 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.500455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:38217 (local address 127.28.210.67:34407)
0806 12:23:21.501141 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:21.501145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:21.501739 (+ 594us) server_negotiation.cc:407] Connection header received
0806 12:23:21.501997 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:21.502001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:21.502053 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:21.502099 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:21.502663 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.503190 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.504636 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.505238 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.505393 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.505458 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:21.505844 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 12:23:21.505940 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593}
I20260806 12:23:21.506551 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.500269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:34407 (local address 127.28.210.65:38217)
0806 12:23:21.501398 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:21.501413 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:21.501853 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:21.502380 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:21.502383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:21.502404 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:21.502518 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.502524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.503324 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.503328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.504430 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.504453 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.505549 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.505553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.505665 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.505697 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 12:23:21.506244 (+ 547us) client_negotiation.cc:240] Negotiation successful
0806 12:23:21.506369 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":399,"thread_start_us":408,"threads_started":1}
I20260806 12:23:21.506755 31138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.500414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:33511 (local address 127.28.210.66:33225)
0806 12:23:21.500564 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:21.500571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:21.501496 (+ 925us) server_negotiation.cc:407] Connection header received
0806 12:23:21.501722 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:21.501725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:21.501778 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:21.501825 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:21.502402 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.503170 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.505303 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.505893 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.506013 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.506066 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:21.506529 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 12:23:21.506623 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 12:23:21.507215 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.500269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:33225 (local address 127.28.210.65:33511)
0806 12:23:21.501267 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:21.501281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:21.501592 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:21.501991 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:21.501993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:21.502014 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:21.502264 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.502274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.503339 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.503343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.505123 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:21.505136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.506175 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.506178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:21.506363 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.506391 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 12:23:21.506911 (+ 520us) client_negotiation.cc:240] Negotiation successful
0806 12:23:21.507020 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":1,"thread_start_us":614,"threads_started":1}
I20260806 12:23:21.507329 31038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb8719f20614415b0382b83cbeb8d20" candidate_uuid: "7576ca6f412545248f9546d5ecf018b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d9f74eb829414f8a534846027efe47" is_pre_election: true
I20260806 12:23:21.507485 31040 raft_consensus.cc:2454] T ecb8719f20614415b0382b83cbeb8d20 P 83d9f74eb829414f8a534846027efe47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a7b20728f44d4d8ada788548ab74ea in term 1.
I20260806 12:23:21.507905 30637 raft_consensus.cc:2454] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71a7b20728f44d4d8ada788548ab74ea in term 1.
I20260806 12:23:21.507969 30839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb8719f20614415b0382b83cbeb8d20" candidate_uuid: "7576ca6f412545248f9546d5ecf018b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a7b20728f44d4d8ada788548ab74ea" is_pre_election: true
I20260806 12:23:21.508322 30747 leader_election.cc:302] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a7b20728f44d4d8ada788548ab74ea, 7576ca6f412545248f9546d5ecf018b1; no voters:
I20260806 12:23:21.508363 30839 raft_consensus.cc:2379] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7576ca6f412545248f9546d5ecf018b1 in current term 1: Already voted for candidate 71a7b20728f44d4d8ada788548ab74ea in this term.
I20260806 12:23:21.508608 31151 raft_consensus.cc:2781] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:21.508920 31151 raft_consensus.cc:686] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [term 1 LEADER]: Becoming Leader. State: Replica: 71a7b20728f44d4d8ada788548ab74ea, State: Running, Role: LEADER
I20260806 12:23:21.509123 30535 leader_election.cc:302] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7576ca6f412545248f9546d5ecf018b1; no voters: 71a7b20728f44d4d8ada788548ab74ea, 83d9f74eb829414f8a534846027efe47
I20260806 12:23:21.509356 31151 consensus_queue.cc:237] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d9f74eb829414f8a534846027efe47" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34407 } } peers { permanent_uuid: "7576ca6f412545248f9546d5ecf018b1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34227 } } peers { permanent_uuid: "71a7b20728f44d4d8ada788548ab74ea" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 33225 } }
I20260806 12:23:21.509500 31152 raft_consensus.cc:2726] T ecb8719f20614415b0382b83cbeb8d20 P 7576ca6f412545248f9546d5ecf018b1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 12:23:21.513789 30412 catalog_manager.cc:5495] T ecb8719f20614415b0382b83cbeb8d20 P 71a7b20728f44d4d8ada788548ab74ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 71a7b20728f44d4d8ada788548ab74ea (127.28.210.66). New cstate: current_term: 1 leader_uuid: "71a7b20728f44d4d8ada788548ab74ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83d9f74eb829414f8a534846027efe47" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7576ca6f412545248f9546d5ecf018b1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71a7b20728f44d4d8ada788548ab74ea" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 33225 } health_report { overall_health: HEALTHY } } }
Aug 06 12:23:21 dist-test-slave-lgr5 krb5kdc[30348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.65@KRBTEST.COM
I20260806 12:23:21.553177 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.542655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.28.210.65:34227)
0806 12:23:21.542815 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:21.542821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:21.543180 (+ 359us) server_negotiation.cc:407] Connection header received
0806 12:23:21.543321 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:21.543324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:21.543375 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:21.543446 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:21.544272 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.544804 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.545902 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.546128 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.546217 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.549212 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:21.549232 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:21.549255 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:21.549314 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:21.551120 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:21.551645 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:21.551652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:21.551653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:21.551779 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:21.552120 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:21.552124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:21.552126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:21.552468 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:21.552651 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:21.552924 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 12:23:21.553043 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 12:23:21 dist-test-slave-lgr5 krb5kdc[30348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 12:23:21 dist-test-slave-lgr5 krb5kdc[30348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.210.65@KRBTEST.COM
I20260806 12:23:21.590924 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.579995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51864 (local address 127.28.210.65:34227)
0806 12:23:21.580200 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:21.580206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:21.580442 (+ 236us) server_negotiation.cc:407] Connection header received
0806 12:23:21.580553 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:21.580557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:21.580610 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:21.580689 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:21.581594 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.582157 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.583212 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.583500 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.583617 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.587024 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:21.587029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:21.587031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:21.587073 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:21.588771 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:21.589464 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:21.589468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:21.589470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:21.589577 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:21.589913 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:21.589915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:21.589916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:21.590127 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:21.590253 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:21.590617 (+ 364us) server_negotiation.cc:299] Negotiation successful
0806 12:23:21.590739 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 12:23:21 dist-test-slave-lgr5 krb5kdc[30348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 12:23:21 dist-test-slave-lgr5 krb5kdc[30348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.65@KRBTEST.COM
I20260806 12:23:21.628491 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:21.616651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.28.210.65:34227)
0806 12:23:21.616820 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:21.616826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:21.617556 (+ 730us) server_negotiation.cc:407] Connection header received
0806 12:23:21.617614 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:21.617617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:21.617668 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:21.617750 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:21.618778 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.619364 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.620761 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:21.621029 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:21.621171 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:21.624642 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:21.624647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:21.624649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:21.624692 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:21.626264 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:21.626969 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:21.626974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:21.626976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:21.627091 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:21.627432 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:21.627436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:21.627437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:21.627700 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:21.627839 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:21.628172 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 12:23:21.628303 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 12:23:21.630723 30573 tablet_service.cc:2908] Rejecting scan request for tablet ecb8719f20614415b0382b83cbeb8d20: Uninitialized: clean time has not yet been initialized
I20260806 12:23:21.633970 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 30510
I20260806 12:23:21.648768 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 30723
I20260806 12:23:21.666857 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 30917
I20260806 12:23:21.683779 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 30369
2026-08-06T12:23:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2972 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:23:21 dist-test-slave-lgr5 krb5kdc[31180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:23:21 dist-test-slave-lgr5 krb5kdc[31180](info): set up 2 sockets
Aug 06 12:23:21 dist-test-slave-lgr5 krb5kdc[31180](info): commencing operation
krb5kdc: starting...
W20260806 12:23:22.897874 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:23:22 dist-test-slave-lgr5 krb5kdc[31180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:23:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:23:23.048702 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:40931
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:40931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:23.191738 31199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:23.192173 31199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:23.192255 31199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:23.196702 31199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:23.196950 31199 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 12:23:23.197018 31199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:23.197057 31199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:23.197091 31199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:23.197124 31199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:23.203374 31199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:40931
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:40931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:23.204718 31199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:23.205848 31199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:23.216650 31207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:23.216653 31209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:23.216656 31211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:23.217064 31199 server_base.cc:1034] running on GCE node
I20260806 12:23:23.217756 31199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:23.219152 31199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:23.220404 31199 hybrid_clock.cc:648] HybridClock initialized: now 1786019003220381 us; error 40 us; skew 500 ppm
Aug 06 12:23:23 dist-test-slave-lgr5 krb5kdc[31180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:23.224251 31199 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:23:23.225709 31199 webserver.cc:460] Webserver started at http://127.28.210.126:38441/ using document root <none> and password file <none>
I20260806 12:23:23.226408 31199 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:23.226495 31199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:23.226883 31199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:23.231556 31199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "c91efe66857e49e9af049e80da1b5017"
format_stamp: "Formatted at 2026-08-06 12:23:23 on dist-test-slave-lgr5"
server_key: "8f37c736a6af4ab9fa0c4b936322f44a"
server_key_iv: "107a6cfb370c2ba4f478dc2d00eb066f"
server_key_version: "encryptionkey@0"
I20260806 12:23:23.233419 31199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "c91efe66857e49e9af049e80da1b5017"
format_stamp: "Formatted at 2026-08-06 12:23:23 on dist-test-slave-lgr5"
server_key: "8f37c736a6af4ab9fa0c4b936322f44a"
server_key_iv: "107a6cfb370c2ba4f478dc2d00eb066f"
server_key_version: "encryptionkey@0"
I20260806 12:23:23.238909 31199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 12:23:23.242240 31221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:23.243948 31199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:23:23.244181 31199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "c91efe66857e49e9af049e80da1b5017"
format_stamp: "Formatted at 2026-08-06 12:23:23 on dist-test-slave-lgr5"
server_key: "8f37c736a6af4ab9fa0c4b936322f44a"
server_key_iv: "107a6cfb370c2ba4f478dc2d00eb066f"
server_key_version: "encryptionkey@0"
I20260806 12:23:23.244329 31199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:23.262169 31199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:23.266438 31199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:23.266729 31199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:23.277762 31199 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:40931
I20260806 12:23:23.277801 31320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:40931 every 8 connection(s)
I20260806 12:23:23.278919 31199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:23.282465 31321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:23.289214 31321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017: Bootstrap starting.
I20260806 12:23:23.289315 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 31199
I20260806 12:23:23.289443 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:23.289709 29513 external_mini_cluster.cc:1428] Setting key a51ded1c8c856093d02661b94908de60
I20260806 12:23:23.292228 31321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:23.293066 31321 log.cc:824] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:23.296427 31321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017: No bootstrap required, opened a new log
Aug 06 12:23:23 dist-test-slave-lgr5 krb5kdc[31180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:23.300231 31321 raft_consensus.cc:348] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c91efe66857e49e9af049e80da1b5017" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40931 } }
I20260806 12:23:23.300580 31321 raft_consensus.cc:374] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:23.300644 31321 raft_consensus.cc:729] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91efe66857e49e9af049e80da1b5017, State: Initialized, Role: FOLLOWER
I20260806 12:23:23.301208 31321 consensus_queue.cc:260] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c91efe66857e49e9af049e80da1b5017" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40931 } }
I20260806 12:23:23.301419 31321 raft_consensus.cc:388] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:23.301487 31321 raft_consensus.cc:482] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:23.301604 31321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:23.302919 31321 raft_consensus.cc:504] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c91efe66857e49e9af049e80da1b5017" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40931 } }
I20260806 12:23:23.303360 31321 leader_election.cc:302] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c91efe66857e49e9af049e80da1b5017; no voters:
I20260806 12:23:23.303758 31321 leader_election.cc:288] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:23.303902 31332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:23.304167 31332 raft_consensus.cc:686] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [term 1 LEADER]: Becoming Leader. State: Replica: c91efe66857e49e9af049e80da1b5017, State: Running, Role: LEADER
I20260806 12:23:23.304541 31332 consensus_queue.cc:237] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c91efe66857e49e9af049e80da1b5017" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40931 } }
I20260806 12:23:23.305305 31321 sys_catalog.cc:564] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:23.305979 31332 sys_catalog.cc:455] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c91efe66857e49e9af049e80da1b5017. Latest consensus state: current_term: 1 leader_uuid: "c91efe66857e49e9af049e80da1b5017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c91efe66857e49e9af049e80da1b5017" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40931 } } }
I20260806 12:23:23.306900 31332 sys_catalog.cc:458] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:23.306710 31334 sys_catalog.cc:455] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c91efe66857e49e9af049e80da1b5017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c91efe66857e49e9af049e80da1b5017" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40931 } } }
I20260806 12:23:23.307139 31334 sys_catalog.cc:458] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:23.307322 31345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:23.307799 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:23.291334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33606 (local address 127.28.210.126:40931)
0806 12:23:23.291996 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:23.292020 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:23.292066 (+ 46us) server_negotiation.cc:407] Connection header received
0806 12:23:23.292990 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:23.293013 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:23.293444 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:23.293817 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:23.294777 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.295864 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.296915 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.297648 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.298422 (+ 774us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:23.301116 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:23.301123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:23.301142 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:23.301187 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:23.303499 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:23.304257 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:23.304264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:23.304271 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:23.304469 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:23.304811 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:23.304814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:23.304816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:23.305408 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:23.305606 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:23.305967 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 12:23:23.306379 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":190,"threads_started":1}
I20260806 12:23:23.311296 31345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:23.318747 31345 catalog_manager.cc:1347] Generated new cluster ID: e76adc8571944518828a1eb322a044da
I20260806 12:23:23.318871 31345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:23.341929 31345 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:23.343116 31345 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:23.355964 31345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017: Generated new TSK 0
I20260806 12:23:23.356880 31345 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:23:23.393692 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:40931
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:23.523468 31363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:23.523819 31363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:23.523869 31363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:23.528144 31363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:23.528245 31363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:23.528288 31363 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 12:23:23.528326 31363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:23:23.534261 31363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.210.126:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:23.535562 31363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:23.536841 31363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:23.549871 31374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:23.549849 31371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:23.549839 31372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:23.550994 31363 server_base.cc:1034] running on GCE node
I20260806 12:23:23.551690 31363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:23.552832 31363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:23.554069 31363 hybrid_clock.cc:648] HybridClock initialized: now 1786019003554057 us; error 43 us; skew 500 ppm
Aug 06 12:23:23 dist-test-slave-lgr5 krb5kdc[31180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:23.557858 31363 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:23:23.559149 31363 webserver.cc:460] Webserver started at http://127.28.210.65:34995/ using document root <none> and password file <none>
I20260806 12:23:23.559823 31363 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:23.559904 31363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:23.560184 31363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:23.562682 31363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "be4896862d4c4f9a977036f46c79bf10"
format_stamp: "Formatted at 2026-08-06 12:23:23 on dist-test-slave-lgr5"
server_key: "1ceafc76f666fc42a6886575367d3822"
server_key_iv: "9766ff00fd69bb870c0f31d9062079f4"
server_key_version: "encryptionkey@0"
I20260806 12:23:23.563338 31363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "be4896862d4c4f9a977036f46c79bf10"
format_stamp: "Formatted at 2026-08-06 12:23:23 on dist-test-slave-lgr5"
server_key: "1ceafc76f666fc42a6886575367d3822"
server_key_iv: "9766ff00fd69bb870c0f31d9062079f4"
server_key_version: "encryptionkey@0"
I20260806 12:23:23.567719 31363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 12:23:23.570990 31389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:23.572646 31363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:23.572799 31363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "be4896862d4c4f9a977036f46c79bf10"
format_stamp: "Formatted at 2026-08-06 12:23:23 on dist-test-slave-lgr5"
server_key: "1ceafc76f666fc42a6886575367d3822"
server_key_iv: "9766ff00fd69bb870c0f31d9062079f4"
server_key_version: "encryptionkey@0"
I20260806 12:23:23.572929 31363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:23.592258 31363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:23.596320 31363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:23.596583 31363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:23.598773 31363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:23.598850 31363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:23.598920 31363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:23.598954 31363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 12:23:23 dist-test-slave-lgr5 krb5kdc[31180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:23.617097 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:23.599767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:50391 (local address 127.28.210.126:40931)
0806 12:23:23.599902 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:23.599906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:23.601257 (+ 1351us) server_negotiation.cc:407] Connection header received
0806 12:23:23.602455 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:23.602459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:23.602528 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:23.602592 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:23.604253 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.605278 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.606941 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.607230 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.607373 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:23.611931 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:23.611935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:23.611937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:23.611972 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:23.613940 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:23.614738 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:23.614742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:23.614743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:23.615082 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:23.615615 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:23.615618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:23.615620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:23.615881 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:23.615996 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:23.616718 (+ 722us) server_negotiation.cc:299] Negotiation successful
0806 12:23:23.616906 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 12:23:23.618218 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:23.600483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40931 (local address 127.28.210.65:50391)
0806 12:23:23.601077 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:23.601129 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:23.602153 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:23.603024 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:23.603032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:23.603439 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:23.604027 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:23.604045 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.605472 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.605477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:23.606766 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:23.606784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.607422 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.607426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:23.607650 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:23.608413 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:23.608441 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:23.611661 (+ 3220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:23.614165 (+ 2504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:23.614175 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:23.614198 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:23.614591 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:23.615238 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:23.615242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:23.615244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:23.615487 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:23.616025 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:23.616034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:23.616407 (+ 373us) client_negotiation.cc:769] Sending connection context
0806 12:23:23.616625 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 12:23:23.617088 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":170,"threads_started":1}
I20260806 12:23:23.619309 31363 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:42559
I20260806 12:23:23.619401 31581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:42559 every 8 connection(s)
I20260806 12:23:23.620810 31363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:23:23.622872 31582 heartbeater.cc:347] Connected to a master server at 127.28.210.126:40931
I20260806 12:23:23.623063 31582 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:23.623602 31582 heartbeater.cc:511] Master 127.28.210.126:40931 requested a full tablet report, sending...
I20260806 12:23:23.623628 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 31363
I20260806 12:23:23.623754 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:23:23.624162 29513 external_mini_cluster.cc:1428] Setting key 36c0d65cdc4cd6688ca24f5f1c571208
I20260806 12:23:23.625474 31247 ts_manager.cc:194] Registered new tserver with Master: be4896862d4c4f9a977036f46c79bf10 (127.28.210.65:42559)
I20260806 12:23:23.627195 31247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:50391
I20260806 12:23:23.632833 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:23.623706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:36771 (local address 127.28.210.126:40931)
0806 12:23:23.623851 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:23.623856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:23.624284 (+ 428us) server_negotiation.cc:407] Connection header received
0806 12:23:23.624631 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:23.624634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:23.624699 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:23.624784 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:23.625309 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.625961 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.626975 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.627231 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.627350 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:23.628998 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:23.629003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:23.629006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:23.629039 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:23.630825 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:23.631417 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:23.631421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:23.631422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:23.631514 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:23.631854 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:23.631858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:23.631859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:23.632129 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:23.632235 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:23.632566 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 12:23:23.632692 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 12:23:23.632911 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:23.623617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40931 (local address 127.28.210.65:36771)
0806 12:23:23.624182 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:23.624198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:23.624507 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:23.624813 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:23.624816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:23.625058 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:23.625175 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:23.625182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.626099 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.626102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:23.626744 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:23.626754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.627434 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.627438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:23.627555 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:23.628228 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:23.628242 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:23.628768 (+ 526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:23.630996 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:23.631000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:23.631002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:23.631289 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:23.631598 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:23.631601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:23.631603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:23.631707 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:23.632250 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:23.632253 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:23.632376 (+ 123us) client_negotiation.cc:769] Sending connection context
0806 12:23:23.632501 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 12:23:23.632680 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":192,"threads_started":1}
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:23:23.659952 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:40931
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:23.786513 31598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:23.786861 31598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:23.786933 31598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:23.791199 31598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:23.791316 31598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:23.791383 31598 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 12:23:23.791443 31598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:23:23.797179 31598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.210.126:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:23.798511 31598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:23.799727 31598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:23.810966 31609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:23.810986 31607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:23.811038 31598 server_base.cc:1034] running on GCE node
W20260806 12:23:23.810963 31606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:23.811729 31598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:23.812513 31598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:23.813678 31598 hybrid_clock.cc:648] HybridClock initialized: now 1786019003813644 us; error 48 us; skew 500 ppm
Aug 06 12:23:23 dist-test-slave-lgr5 krb5kdc[31180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:23.816825 31598 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:23:23.817855 31598 webserver.cc:460] Webserver started at http://127.28.210.66:36905/ using document root <none> and password file <none>
I20260806 12:23:23.818440 31598 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:23.818521 31598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:23.818745 31598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:23.821070 31598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "43df296ac3e940ea936e8d31e95817be"
format_stamp: "Formatted at 2026-08-06 12:23:23 on dist-test-slave-lgr5"
server_key: "a74277ea6a68f41b1c044a446fe08f63"
server_key_iv: "3d793670cc775d78874cbd51f7d91f09"
server_key_version: "encryptionkey@0"
I20260806 12:23:23.821661 31598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "43df296ac3e940ea936e8d31e95817be"
format_stamp: "Formatted at 2026-08-06 12:23:23 on dist-test-slave-lgr5"
server_key: "a74277ea6a68f41b1c044a446fe08f63"
server_key_iv: "3d793670cc775d78874cbd51f7d91f09"
server_key_version: "encryptionkey@0"
I20260806 12:23:23.825610 31598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:23:23.828423 31617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:23.829890 31598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:23.830010 31598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "43df296ac3e940ea936e8d31e95817be"
format_stamp: "Formatted at 2026-08-06 12:23:23 on dist-test-slave-lgr5"
server_key: "a74277ea6a68f41b1c044a446fe08f63"
server_key_iv: "3d793670cc775d78874cbd51f7d91f09"
server_key_version: "encryptionkey@0"
I20260806 12:23:23.830123 31598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:23.844748 31598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:23.848364 31598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:23.848583 31598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:23.850409 31598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:23.850466 31598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:23.850510 31598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:23.850551 31598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:23.860889 31598 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:40127
I20260806 12:23:23.860981 31801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:40127 every 8 connection(s)
Aug 06 12:23:23 dist-test-slave-lgr5 krb5kdc[31180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:23.862135 31598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:23:23.866240 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:23.851161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:57537 (local address 127.28.210.126:40931)
0806 12:23:23.851324 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:23.851330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:23.852678 (+ 1348us) server_negotiation.cc:407] Connection header received
0806 12:23:23.853940 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:23.853944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:23.854009 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:23.854087 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:23.855914 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.856816 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.858547 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.858904 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.859047 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:23.862173 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:23.862178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:23.862182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:23.862222 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:23.863869 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:23.864489 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:23.864493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:23.864494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:23.864583 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:23.864950 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:23.864954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:23.864955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:23.865187 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:23.865284 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:23.865897 (+ 613us) server_negotiation.cc:299] Negotiation successful
0806 12:23:23.866052 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 12:23:23.867338 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:23.851990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40931 (local address 127.28.210.66:57537)
0806 12:23:23.852507 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:23.852561 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:23.853691 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:23.854316 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:23.854329 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:23.854840 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:23.855673 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:23.855700 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.857001 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.857005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:23.858370 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:23.858387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.859120 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.859125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:23.859366 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:23.860056 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:23.860075 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:23.861915 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:23.864017 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:23.864024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:23.864042 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:23.864366 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:23.864679 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:23.864683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:23.864685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:23.864840 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:23.865304 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:23.865312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:23.865622 (+ 310us) client_negotiation.cc:769] Sending connection context
0806 12:23:23.865808 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 12:23:23.866219 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":109,"threads_started":1}
I20260806 12:23:23.868692 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 31598
I20260806 12:23:23.868819 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:23:23.868825 31802 heartbeater.cc:347] Connected to a master server at 127.28.210.126:40931
I20260806 12:23:23.869045 31802 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:23.869321 29513 external_mini_cluster.cc:1428] Setting key 8d685dc04042de31362e606e45caa549
I20260806 12:23:23.869690 31802 heartbeater.cc:511] Master 127.28.210.126:40931 requested a full tablet report, sending...
I20260806 12:23:23.870935 31247 ts_manager.cc:194] Registered new tserver with Master: 43df296ac3e940ea936e8d31e95817be (127.28.210.66:40127)
I20260806 12:23:23.871762 31247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:57537
I20260806 12:23:23.880610 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:23.871902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:57671 (local address 127.28.210.126:40931)
0806 12:23:23.872040 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:23.872045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:23.872555 (+ 510us) server_negotiation.cc:407] Connection header received
0806 12:23:23.872660 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:23.872663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:23.872710 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:23.872801 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:23.873439 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.874174 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.875052 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.875279 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.875376 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:23.876888 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:23.876892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:23.876894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:23.876931 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:23.878603 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:23.879168 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:23.879172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:23.879174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:23.879268 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:23.879542 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:23.879545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:23.879547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:23.879809 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:23.879911 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:23.880270 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 12:23:23.880400 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 12:23:23.880615 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:23.871980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40931 (local address 127.28.210.66:57671)
0806 12:23:23.872442 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:23.872458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:23.872579 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:23.872828 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:23.872830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:23.873081 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:23.873222 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:23.873230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.874301 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.874305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:23.874908 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:23.874919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:23.875397 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:23.875401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:23.875501 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:23.876123 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:23.876143 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:23.876664 (+ 521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:23.878787 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:23.878790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:23.878791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:23.879064 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:23.879360 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:23.879364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:23.879366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:23.879450 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:23.879913 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:23.879917 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:23.880035 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 12:23:23.880202 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 12:23:23.880401 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":247,"threads_started":1}
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:23:23.907670 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:40931
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:24.045600 31816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:24.046011 31816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:24.046083 31816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:24.050544 31816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:24.050635 31816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:24.050695 31816 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 12:23:24.050756 31816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:23:24.056970 31816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.210.126:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:24.058436 31816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:24.059669 31816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:24.072849 31828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:24.072710 31825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:24.072716 31824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:24.073037 31816 server_base.cc:1034] running on GCE node
I20260806 12:23:24.073556 31816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:24.074602 31816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:24.075887 31816 hybrid_clock.cc:648] HybridClock initialized: now 1786019004075803 us; error 80 us; skew 500 ppm
Aug 06 12:23:24 dist-test-slave-lgr5 krb5kdc[31180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019004, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:24.079710 31816 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:23:24.080991 31816 webserver.cc:460] Webserver started at http://127.28.210.67:37963/ using document root <none> and password file <none>
I20260806 12:23:24.081668 31816 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:24.081749 31816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:24.081998 31816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:24.085225 31816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "06898be4f939488fb08e4943b09c3318"
format_stamp: "Formatted at 2026-08-06 12:23:24 on dist-test-slave-lgr5"
server_key: "3a6c0996b7c54b066b03bf2936e7ab47"
server_key_iv: "790d2445ec5684216b1f43e98fdfd526"
server_key_version: "encryptionkey@0"
I20260806 12:23:24.086009 31816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "06898be4f939488fb08e4943b09c3318"
format_stamp: "Formatted at 2026-08-06 12:23:24 on dist-test-slave-lgr5"
server_key: "3a6c0996b7c54b066b03bf2936e7ab47"
server_key_iv: "790d2445ec5684216b1f43e98fdfd526"
server_key_version: "encryptionkey@0"
I20260806 12:23:24.091060 31816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 12:23:24.095045 31836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:24.096628 31816 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 12:23:24.096791 31816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "06898be4f939488fb08e4943b09c3318"
format_stamp: "Formatted at 2026-08-06 12:23:24 on dist-test-slave-lgr5"
server_key: "3a6c0996b7c54b066b03bf2936e7ab47"
server_key_iv: "790d2445ec5684216b1f43e98fdfd526"
server_key_version: "encryptionkey@0"
I20260806 12:23:24.096930 31816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:24.116925 31816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:24.121145 31816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:24.121387 31816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:24.123608 31816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:24.123682 31816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:24.123734 31816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:24.123754 31816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 12:23:24 dist-test-slave-lgr5 krb5kdc[31180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019004, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:24.141804 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.124746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:44463 (local address 127.28.210.126:40931)
0806 12:23:24.124891 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.124895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.126371 (+ 1476us) server_negotiation.cc:407] Connection header received
0806 12:23:24.127697 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.127702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.127781 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.127906 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:24.129526 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.130408 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.132304 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.132547 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.132656 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.137045 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:24.137052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:24.137055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:24.137094 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:24.139273 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:24.139932 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:24.139937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:24.139939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:24.140041 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:24.140523 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:24.140527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:24.140529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:24.140755 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:24.140851 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.141465 (+ 614us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.141624 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 12:23:24.142926 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.125443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40931 (local address 127.28.210.67:44463)
0806 12:23:24.126130 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:24.126186 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:24.127228 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:24.128152 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:24.128164 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:24.128734 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:24.129318 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.129337 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.130587 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.130591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.132041 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.132055 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.132725 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.132730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.132937 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.133876 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:24.133898 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:24.136680 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:24.139434 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:24.139441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:24.139458 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:24.139776 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:24.140160 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:24.140164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:24.140166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:24.140388 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:24.140869 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:24.140877 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:24.141191 (+ 314us) client_negotiation.cc:769] Sending connection context
0806 12:23:24.141380 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 12:23:24.141791 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":153,"threads_started":1}
I20260806 12:23:24.143280 31816 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:38715
I20260806 12:23:24.143654 32001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:38715 every 8 connection(s)
I20260806 12:23:24.144552 31816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:23:24.146183 32002 heartbeater.cc:347] Connected to a master server at 127.28.210.126:40931
I20260806 12:23:24.146364 32002 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:24.146960 32002 heartbeater.cc:511] Master 127.28.210.126:40931 requested a full tablet report, sending...
I20260806 12:23:24.147972 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 31816
I20260806 12:23:24.148137 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:23:24.148378 31247 ts_manager.cc:194] Registered new tserver with Master: 06898be4f939488fb08e4943b09c3318 (127.28.210.67:38715)
I20260806 12:23:24.148496 29513 external_mini_cluster.cc:1428] Setting key 104623bc9def612c412995031ccd816d
I20260806 12:23:24.149219 31247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:44463
I20260806 12:23:24.151316 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:23:24.158100 32011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.147142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40931 (local address 127.28.210.67:43195)
0806 12:23:24.147574 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:24.147586 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:24.147968 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:24.148426 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:24.148429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:24.148669 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:24.148814 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.148822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.151810 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.151812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.152456 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.152468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.153131 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.153134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.153223 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.153948 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:24.153961 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:24.154330 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:24.156498 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:24.156502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:24.156504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:24.156722 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:24.157002 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:24.157005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:24.157007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:24.157094 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:24.157496 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:24.157500 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:24.157601 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 12:23:24.157694 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 12:23:24.157883 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":112,"threads_started":1}
I20260806 12:23:24.158087 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.147227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:43195 (local address 127.28.210.126:40931)
0806 12:23:24.147394 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.147399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.147861 (+ 462us) server_negotiation.cc:407] Connection header received
0806 12:23:24.148134 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.148138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.148198 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.148256 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:24.148941 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.149696 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.152595 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.152958 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.153083 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.154514 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:24.154518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:24.154527 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:24.154559 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:24.156375 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:24.156824 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:24.156828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:24.156830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:24.156914 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:24.157182 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:24.157185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:24.157187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:24.157399 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:24.157492 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.157770 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.157897 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 12:23:24 dist-test-slave-lgr5 krb5kdc[31180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 12:23:24 dist-test-slave-lgr5 krb5kdc[31180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:24.183835 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.175176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.28.210.126:40931)
0806 12:23:24.175359 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.175365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.175383 (+ 18us) server_negotiation.cc:407] Connection header received
0806 12:23:24.175545 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.175547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.175595 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.175672 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:24.176473 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.177049 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.177946 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.178175 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.178269 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.180856 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:24.180860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:24.180862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:24.180894 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:24.182170 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:24.182604 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:24.182607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:24.182608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:24.182691 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:24.182947 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:24.182950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:24.182952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:24.183160 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:24.183299 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.183542 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.183667 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 12:23:24.187633 31247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:23:24.191116 31247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:24.211035 32030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.203390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.28.210.67:38715)
0806 12:23:24.203976 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.203989 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.204027 (+ 38us) server_negotiation.cc:407] Connection header received
0806 12:23:24.204206 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.204213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.204399 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.204633 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:24.205120 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.206400 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.208913 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.209881 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.210081 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.210358 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.210654 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.210797 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":177,"threads_started":1}
I20260806 12:23:24.211176 32028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.202455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:38715 (local address 127.0.0.1:53792)
0806 12:23:24.203816 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:24.203833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:24.204041 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:24.204814 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:24.204817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:24.204838 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:24.204979 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.204987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.206545 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.206549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.208628 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.208663 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.210151 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.210157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.210347 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.210419 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 12:23:24.210577 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 12:23:24.210801 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1268,"mutex_wait_us":158,"thread_start_us":182,"threads_started":1}
I20260806 12:23:24.212637 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.203304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:40127 (local address 127.0.0.1:49424)
0806 12:23:24.204128 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:24.204142 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:24.204493 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:24.205143 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:24.205146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:24.205167 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:24.205248 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.205253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.206655 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.206658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.208613 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.208624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.212184 (+ 3560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.212188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.212277 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.212298 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 12:23:24.212377 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 12:23:24.212466 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":112,"thread_start_us":216,"threads_started":1}
I20260806 12:23:24.212652 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.202354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:42559 (local address 127.0.0.1:45434)
0806 12:23:24.202857 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:24.202915 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:24.203067 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:24.204666 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:24.204673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:24.204699 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:24.204876 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.204886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.206546 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.206549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.208580 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.208605 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.212023 (+ 3418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.212027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.212202 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.212237 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 12:23:24.212325 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 12:23:24.212435 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":75,"thread_start_us":177,"threads_started":1}
I20260806 12:23:24.213925 32031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.203513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.28.210.65:42559)
0806 12:23:24.203953 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.203962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.203993 (+ 31us) server_negotiation.cc:407] Connection header received
0806 12:23:24.204122 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.204128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.204427 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.204683 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:24.205014 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.206400 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.210527 (+ 4127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.211822 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.211980 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.213278 (+ 1298us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.213359 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.213619 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":83,"threads_started":1}
I20260806 12:23:24.215785 31909 tablet_service.cc:1467] Processing CreateTablet for tablet ee84ef3490ee4a3e8ef68d03dfd436f6 (DEFAULT_TABLE table=test-table [id=72a5fd9dc0ca4191ae6dc476a28a3408]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:24.217402 31909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee84ef3490ee4a3e8ef68d03dfd436f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:24.217515 31470 tablet_service.cc:1467] Processing CreateTablet for tablet ee84ef3490ee4a3e8ef68d03dfd436f6 (DEFAULT_TABLE table=test-table [id=72a5fd9dc0ca4191ae6dc476a28a3408]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:24.218701 31470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee84ef3490ee4a3e8ef68d03dfd436f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:24.221988 32034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.203481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.28.210.66:40127)
0806 12:23:24.203953 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.203965 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.204490 (+ 525us) server_negotiation.cc:407] Connection header received
0806 12:23:24.204610 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.204615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.204812 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.205198 (+ 386us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:24.205372 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.206514 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.208821 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.209964 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.210133 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.210398 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.212510 (+ 2112us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.221826 (+ 9316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":174,"threads_started":1}
I20260806 12:23:24.225659 32036 tablet_bootstrap.cc:492] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10: Bootstrap starting.
I20260806 12:23:24.225555 31700 tablet_service.cc:1467] Processing CreateTablet for tablet ee84ef3490ee4a3e8ef68d03dfd436f6 (DEFAULT_TABLE table=test-table [id=72a5fd9dc0ca4191ae6dc476a28a3408]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:24.226938 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee84ef3490ee4a3e8ef68d03dfd436f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:24.227645 32038 tablet_bootstrap.cc:492] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318: Bootstrap starting.
I20260806 12:23:24.228593 32036 tablet_bootstrap.cc:654] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:24.229560 32036 log.cc:824] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:24.230064 32038 tablet_bootstrap.cc:654] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:24.230916 32038 log.cc:824] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:24.232240 32036 tablet_bootstrap.cc:492] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10: No bootstrap required, opened a new log
I20260806 12:23:24.232487 32036 ts_tablet_manager.cc:1397] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 12:23:24.232827 32038 tablet_bootstrap.cc:492] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318: No bootstrap required, opened a new log
I20260806 12:23:24.233022 32038 ts_tablet_manager.cc:1397] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 12:23:24.237193 32038 raft_consensus.cc:348] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } }
I20260806 12:23:24.237586 32038 raft_consensus.cc:374] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:24.237658 32038 raft_consensus.cc:729] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06898be4f939488fb08e4943b09c3318, State: Initialized, Role: FOLLOWER
I20260806 12:23:24.237838 32036 raft_consensus.cc:348] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } }
I20260806 12:23:24.238346 32036 raft_consensus.cc:374] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:24.238252 32038 consensus_queue.cc:260] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } }
I20260806 12:23:24.238432 32036 raft_consensus.cc:729] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4896862d4c4f9a977036f46c79bf10, State: Initialized, Role: FOLLOWER
I20260806 12:23:24.239069 32044 tablet_bootstrap.cc:492] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be: Bootstrap starting.
I20260806 12:23:24.239065 32036 consensus_queue.cc:260] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } }
I20260806 12:23:24.239751 32038 ts_tablet_manager.cc:1428] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 12:23:24.240232 32002 heartbeater.cc:503] Master 127.28.210.126:40931 was elected leader, sending a full tablet report...
I20260806 12:23:24.240415 31582 heartbeater.cc:503] Master 127.28.210.126:40931 was elected leader, sending a full tablet report...
I20260806 12:23:24.240732 32036 ts_tablet_manager.cc:1428] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 12:23:24.242265 32044 tablet_bootstrap.cc:654] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:24.243186 32044 log.cc:824] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:24.246304 32044 tablet_bootstrap.cc:492] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be: No bootstrap required, opened a new log
I20260806 12:23:24.246934 32044 ts_tablet_manager.cc:1397] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260806 12:23:24.251901 32044 raft_consensus.cc:348] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } }
I20260806 12:23:24.252334 32044 raft_consensus.cc:374] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:24.252404 32044 raft_consensus.cc:729] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43df296ac3e940ea936e8d31e95817be, State: Initialized, Role: FOLLOWER
I20260806 12:23:24.253077 32044 consensus_queue.cc:260] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } }
I20260806 12:23:24.254029 31802 heartbeater.cc:503] Master 127.28.210.126:40931 was elected leader, sending a full tablet report...
I20260806 12:23:24.254387 32044 ts_tablet_manager.cc:1428] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 12:23:24.304531 32046 raft_consensus.cc:482] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:24.304816 32046 raft_consensus.cc:504] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } }
I20260806 12:23:24.306308 32046 leader_election.cc:288] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06898be4f939488fb08e4943b09c3318 (127.28.210.67:38715), 43df296ac3e940ea936e8d31e95817be (127.28.210.66:40127)
I20260806 12:23:24.311453 32030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.307133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:53071 (local address 127.28.210.67:38715)
0806 12:23:24.307279 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.307283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.307301 (+ 18us) server_negotiation.cc:407] Connection header received
0806 12:23:24.307395 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.307398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.307450 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.307524 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:24.307882 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.308658 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.310290 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.310849 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.310992 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.311055 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.311256 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.311335 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 12:23:24.311496 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.306670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:38715 (local address 127.28.210.65:53071)
0806 12:23:24.307118 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:24.307135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:24.307296 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:24.307581 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:24.307584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:24.307610 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:24.307747 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.307755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.308785 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.308788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.310144 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.310160 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.311008 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.311017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.311103 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.311128 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 12:23:24.311222 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 12:23:24.311302 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":35,"thread_start_us":159,"threads_started":1}
I20260806 12:23:24.311832 32034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.306866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:32789 (local address 127.28.210.66:40127)
0806 12:23:24.307164 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.307185 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.307871 (+ 686us) server_negotiation.cc:407] Connection header received
0806 12:23:24.307982 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.307985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.308038 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.308154 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:24.308453 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.309072 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.310504 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.311081 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.311179 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.311227 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.311536 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.311616 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260806 12:23:24.311925 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.306710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:40127 (local address 127.28.210.65:32789)
0806 12:23:24.307724 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:24.307739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:24.307885 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:24.308206 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:24.308208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:24.308230 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:24.308341 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.308347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.309190 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.309194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.310383 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.310391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.311206 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.311209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.311365 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.311388 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 12:23:24.311464 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 12:23:24.311527 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":25,"thread_start_us":273,"threads_started":1}
I20260806 12:23:24.312400 31937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee84ef3490ee4a3e8ef68d03dfd436f6" candidate_uuid: "be4896862d4c4f9a977036f46c79bf10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06898be4f939488fb08e4943b09c3318" is_pre_election: true
I20260806 12:23:24.312634 31729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee84ef3490ee4a3e8ef68d03dfd436f6" candidate_uuid: "be4896862d4c4f9a977036f46c79bf10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43df296ac3e940ea936e8d31e95817be" is_pre_election: true
I20260806 12:23:24.312799 31937 raft_consensus.cc:2454] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be4896862d4c4f9a977036f46c79bf10 in term 0.
I20260806 12:23:24.312945 31729 raft_consensus.cc:2454] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be4896862d4c4f9a977036f46c79bf10 in term 0.
I20260806 12:23:24.313395 31390 leader_election.cc:302] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06898be4f939488fb08e4943b09c3318, 43df296ac3e940ea936e8d31e95817be, be4896862d4c4f9a977036f46c79bf10; no voters:
I20260806 12:23:24.313691 32046 raft_consensus.cc:2781] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:23:24.313781 32046 raft_consensus.cc:482] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:24.313832 32046 raft_consensus.cc:3037] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:24.315030 32046 raft_consensus.cc:504] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } }
I20260806 12:23:24.315490 32046 leader_election.cc:288] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [CANDIDATE]: Term 1 election: Requested vote from peers 06898be4f939488fb08e4943b09c3318 (127.28.210.67:38715), 43df296ac3e940ea936e8d31e95817be (127.28.210.66:40127)
I20260806 12:23:24.316170 31729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee84ef3490ee4a3e8ef68d03dfd436f6" candidate_uuid: "be4896862d4c4f9a977036f46c79bf10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43df296ac3e940ea936e8d31e95817be"
I20260806 12:23:24.316339 31729 raft_consensus.cc:3037] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:24.316361 31937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee84ef3490ee4a3e8ef68d03dfd436f6" candidate_uuid: "be4896862d4c4f9a977036f46c79bf10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06898be4f939488fb08e4943b09c3318"
I20260806 12:23:24.316720 31937 raft_consensus.cc:3037] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:24.318113 31729 raft_consensus.cc:2454] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be4896862d4c4f9a977036f46c79bf10 in term 1.
I20260806 12:23:24.318709 31937 raft_consensus.cc:2454] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be4896862d4c4f9a977036f46c79bf10 in term 1.
I20260806 12:23:24.318734 31391 leader_election.cc:302] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43df296ac3e940ea936e8d31e95817be, be4896862d4c4f9a977036f46c79bf10; no voters:
I20260806 12:23:24.319036 32046 raft_consensus.cc:2781] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:24.319434 32046 raft_consensus.cc:686] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 1 LEADER]: Becoming Leader. State: Replica: be4896862d4c4f9a977036f46c79bf10, State: Running, Role: LEADER
I20260806 12:23:24.319883 32046 consensus_queue.cc:237] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } }
I20260806 12:23:24.324389 31246 catalog_manager.cc:5495] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 reported cstate change: term changed from 0 to 1, leader changed from <none> to be4896862d4c4f9a977036f46c79bf10 (127.28.210.65). New cstate: current_term: 1 leader_uuid: "be4896862d4c4f9a977036f46c79bf10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } health_report { overall_health: UNKNOWN } } }
I20260806 12:23:24.351550 32031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.346985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.28.210.65:42559)
0806 12:23:24.347151 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.347156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.347177 (+ 21us) server_negotiation.cc:407] Connection header received
0806 12:23:24.347590 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.347594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.347665 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.347769 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:24.348131 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.348769 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.349792 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.350021 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.350128 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.350528 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:24.351025 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:24.351148 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.351284 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.351379 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 12:23:24.359748 31729 raft_consensus.cc:1261] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be [term 1 FOLLOWER]: Refusing update from remote peer be4896862d4c4f9a977036f46c79bf10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:24.360954 32054 consensus_queue.cc:1035] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:24.361120 31937 raft_consensus.cc:1261] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318 [term 1 FOLLOWER]: Refusing update from remote peer be4896862d4c4f9a977036f46c79bf10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:24.361730 32054 consensus_queue.cc:1035] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 12:23:24.363438 31803 tablet.cc:2369] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:23:24.372430 32063 mvcc.cc:205] Tried to move back new op lower bound from 7315533841846206464 to 7315533841698127872. Current Snapshot: MvccSnapshot[applied={T|T < 7315533841846206464}]
I20260806 12:23:24.388932 31247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:24.389179 31247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:24.392344 31247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017: Sending DeleteTablet for 3 replicas of tablet ee84ef3490ee4a3e8ef68d03dfd436f6
I20260806 12:23:24.393407 31909 tablet_service.cc:1514] Processing DeleteTablet for tablet ee84ef3490ee4a3e8ef68d03dfd436f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:24 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:53792
I20260806 12:23:24.393733 31470 tablet_service.cc:1514] Processing DeleteTablet for tablet ee84ef3490ee4a3e8ef68d03dfd436f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:24 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:45434
I20260806 12:23:24.394240 32071 tablet_replica.cc:330] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318: stopping tablet replica
I20260806 12:23:24.394290 31700 tablet_service.cc:1514] Processing DeleteTablet for tablet ee84ef3490ee4a3e8ef68d03dfd436f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:24 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:49424
I20260806 12:23:24.394394 31247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 12:23:24.394655 32071 raft_consensus.cc:2229] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 12:23:24.394969 31247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:24.395117 32071 raft_consensus.cc:2258] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:24.395339 32073 tablet_replica.cc:330] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be: stopping tablet replica
I20260806 12:23:24.395776 32072 tablet_replica.cc:330] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10: stopping tablet replica
I20260806 12:23:24.396173 32072 raft_consensus.cc:2229] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:23:24.396698 32072 raft_consensus.cc:2258] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:24.397212 32073 raft_consensus.cc:2229] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:24.397589 32073 raft_consensus.cc:2258] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:24.398124 32071 ts_tablet_manager.cc:1905] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:24.399400 32073 ts_tablet_manager.cc:1905] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:24.401754 31909 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4d758afa3c4bbab61d8ab0846678b0 (DEFAULT_TABLE table=test-table [id=dfad2d7f578948b4a8e698661cc06481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:24.401804 31470 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4d758afa3c4bbab61d8ab0846678b0 (DEFAULT_TABLE table=test-table [id=dfad2d7f578948b4a8e698661cc06481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:24.401862 31700 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4d758afa3c4bbab61d8ab0846678b0 (DEFAULT_TABLE table=test-table [id=dfad2d7f578948b4a8e698661cc06481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:24.402072 31909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4d758afa3c4bbab61d8ab0846678b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:24.402083 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4d758afa3c4bbab61d8ab0846678b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:24.402122 31470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4d758afa3c4bbab61d8ab0846678b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:24.404510 32044 tablet_bootstrap.cc:492] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be: Bootstrap starting.
I20260806 12:23:24.404860 32036 tablet_bootstrap.cc:492] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10: Bootstrap starting.
I20260806 12:23:24.404884 32073 ts_tablet_manager.cc:1918] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:23:24.405037 32073 log.cc:1196] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/ee84ef3490ee4a3e8ef68d03dfd436f6
I20260806 12:23:24.405625 32044 tablet_bootstrap.cc:654] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:24.406049 32036 tablet_bootstrap.cc:654] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:24.407506 32036 tablet_bootstrap.cc:492] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10: No bootstrap required, opened a new log
I20260806 12:23:24.407611 32036 ts_tablet_manager.cc:1397] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:23:24.407703 32071 ts_tablet_manager.cc:1918] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:23:24.407835 32071 log.cc:1196] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/ee84ef3490ee4a3e8ef68d03dfd436f6
I20260806 12:23:24.408207 32071 ts_tablet_manager.cc:1939] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 06898be4f939488fb08e4943b09c3318: Deleting consensus metadata
I20260806 12:23:24.408170 32036 raft_consensus.cc:348] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.408324 32036 raft_consensus.cc:374] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:24.408370 32036 raft_consensus.cc:729] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4896862d4c4f9a977036f46c79bf10, State: Initialized, Role: FOLLOWER
I20260806 12:23:24.408514 32036 consensus_queue.cc:260] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.408805 32036 ts_tablet_manager.cc:1428] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 12:23:24.409303 32072 ts_tablet_manager.cc:1905] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:24.409754 32038 tablet_bootstrap.cc:492] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318: Bootstrap starting.
I20260806 12:23:24.410017 32044 tablet_bootstrap.cc:492] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be: No bootstrap required, opened a new log
I20260806 12:23:24.410131 32044 ts_tablet_manager.cc:1397] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 12:23:24.410753 32038 tablet_bootstrap.cc:654] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:24.410693 32044 raft_consensus.cc:348] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.410840 32044 raft_consensus.cc:374] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:24.410890 32044 raft_consensus.cc:729] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43df296ac3e940ea936e8d31e95817be, State: Initialized, Role: FOLLOWER
I20260806 12:23:24.411162 32044 consensus_queue.cc:260] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.411511 32044 ts_tablet_manager.cc:1428] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 12:23:24.411623 31222 catalog_manager.cc:4841] TS 06898be4f939488fb08e4943b09c3318 (127.28.210.67:38715): tablet ee84ef3490ee4a3e8ef68d03dfd436f6 (table test-table [id=72a5fd9dc0ca4191ae6dc476a28a3408]) successfully deleted
I20260806 12:23:24.411993 32073 ts_tablet_manager.cc:1939] T ee84ef3490ee4a3e8ef68d03dfd436f6 P 43df296ac3e940ea936e8d31e95817be: Deleting consensus metadata
I20260806 12:23:24.413501 31223 catalog_manager.cc:4841] TS 43df296ac3e940ea936e8d31e95817be (127.28.210.66:40127): tablet ee84ef3490ee4a3e8ef68d03dfd436f6 (table test-table [id=72a5fd9dc0ca4191ae6dc476a28a3408]) successfully deleted
I20260806 12:23:24.414747 32072 ts_tablet_manager.cc:1918] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:23:24.414867 32072 log.cc:1196] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/wal/wals/ee84ef3490ee4a3e8ef68d03dfd436f6
I20260806 12:23:24.415186 32038 tablet_bootstrap.cc:492] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318: No bootstrap required, opened a new log
I20260806 12:23:24.415206 32072 ts_tablet_manager.cc:1939] T ee84ef3490ee4a3e8ef68d03dfd436f6 P be4896862d4c4f9a977036f46c79bf10: Deleting consensus metadata
I20260806 12:23:24.415295 32038 ts_tablet_manager.cc:1397] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 12:23:24.415788 32038 raft_consensus.cc:348] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.415920 32038 raft_consensus.cc:374] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:24.415961 32038 raft_consensus.cc:729] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06898be4f939488fb08e4943b09c3318, State: Initialized, Role: FOLLOWER
I20260806 12:23:24.416143 32038 consensus_queue.cc:260] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.416424 32038 ts_tablet_manager.cc:1428] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 12:23:24.416507 31223 catalog_manager.cc:4841] TS be4896862d4c4f9a977036f46c79bf10 (127.28.210.65:42559): tablet ee84ef3490ee4a3e8ef68d03dfd436f6 (table test-table [id=72a5fd9dc0ca4191ae6dc476a28a3408]) successfully deleted
I20260806 12:23:24.474148 32048 raft_consensus.cc:482] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:24.474320 32048 raft_consensus.cc:504] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.475603 32048 leader_election.cc:288] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be4896862d4c4f9a977036f46c79bf10 (127.28.210.65:42559), 06898be4f939488fb08e4943b09c3318 (127.28.210.67:38715)
I20260806 12:23:24.481480 32030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.476418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:41331 (local address 127.28.210.67:38715)
0806 12:23:24.476550 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.476554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.476705 (+ 151us) server_negotiation.cc:407] Connection header received
0806 12:23:24.477092 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.477095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.477136 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.477200 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:24.477522 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.478062 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.479920 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.480615 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.480720 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.480779 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.481269 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.481349 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 12:23:24.481521 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.475860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:42559 (local address 127.28.210.66:41485)
0806 12:23:24.476356 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:24.476373 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:24.476670 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:24.476987 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:24.476990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:24.477010 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:24.477151 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.477158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.478241 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.478244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.479769 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.479778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.480915 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.480918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.481105 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.481139 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 12:23:24.481221 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 12:23:24.481313 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":155,"thread_start_us":312,"threads_started":1}
I20260806 12:23:24.481542 32031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.476050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:41485 (local address 127.28.210.65:42559)
0806 12:23:24.476354 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.476360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.476599 (+ 239us) server_negotiation.cc:407] Connection header received
0806 12:23:24.476844 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.476847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.476895 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.476955 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:24.477288 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.478140 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.479920 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.480615 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.480720 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.480779 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.481263 (+ 484us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.481366 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260806 12:23:24.482280 32081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.475921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:38715 (local address 127.28.210.66:41331)
0806 12:23:24.476559 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:24.476572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:24.476742 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:24.477256 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:24.477259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:24.477276 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:24.477386 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.477391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.478188 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.478192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.479765 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.479778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.480915 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.480918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.481100 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.481148 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 12:23:24.482022 (+ 874us) client_negotiation.cc:240] Negotiation successful
0806 12:23:24.482114 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":189,"thread_start_us":92,"threads_started":1}
I20260806 12:23:24.482486 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4d758afa3c4bbab61d8ab0846678b0" candidate_uuid: "43df296ac3e940ea936e8d31e95817be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be4896862d4c4f9a977036f46c79bf10" is_pre_election: true
I20260806 12:23:24.482810 31494 raft_consensus.cc:2454] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43df296ac3e940ea936e8d31e95817be in term 0.
I20260806 12:23:24.483350 31622 leader_election.cc:302] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43df296ac3e940ea936e8d31e95817be, be4896862d4c4f9a977036f46c79bf10; no voters:
I20260806 12:23:24.483387 31937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4d758afa3c4bbab61d8ab0846678b0" candidate_uuid: "43df296ac3e940ea936e8d31e95817be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06898be4f939488fb08e4943b09c3318" is_pre_election: true
I20260806 12:23:24.483554 31937 raft_consensus.cc:2454] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43df296ac3e940ea936e8d31e95817be in term 0.
I20260806 12:23:24.483722 32048 raft_consensus.cc:2781] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:23:24.483924 32048 raft_consensus.cc:482] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:24.483992 32048 raft_consensus.cc:3037] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:24.485257 32048 raft_consensus.cc:504] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.485721 32048 leader_election.cc:288] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [CANDIDATE]: Term 1 election: Requested vote from peers be4896862d4c4f9a977036f46c79bf10 (127.28.210.65:42559), 06898be4f939488fb08e4943b09c3318 (127.28.210.67:38715)
I20260806 12:23:24.486186 31937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4d758afa3c4bbab61d8ab0846678b0" candidate_uuid: "43df296ac3e940ea936e8d31e95817be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06898be4f939488fb08e4943b09c3318"
I20260806 12:23:24.486340 31937 raft_consensus.cc:3037] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:24.487677 31937 raft_consensus.cc:2454] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43df296ac3e940ea936e8d31e95817be in term 1.
I20260806 12:23:24.488011 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4d758afa3c4bbab61d8ab0846678b0" candidate_uuid: "43df296ac3e940ea936e8d31e95817be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be4896862d4c4f9a977036f46c79bf10"
I20260806 12:23:24.488107 31618 leader_election.cc:302] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06898be4f939488fb08e4943b09c3318, 43df296ac3e940ea936e8d31e95817be; no voters:
I20260806 12:23:24.488189 31494 raft_consensus.cc:3037] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:24.488345 32048 raft_consensus.cc:2781] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:24.488600 32048 raft_consensus.cc:686] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 1 LEADER]: Becoming Leader. State: Replica: 43df296ac3e940ea936e8d31e95817be, State: Running, Role: LEADER
I20260806 12:23:24.488906 32048 consensus_queue.cc:237] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.489395 31494 raft_consensus.cc:2454] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43df296ac3e940ea936e8d31e95817be in term 1.
I20260806 12:23:24.491852 31245 catalog_manager.cc:5495] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be reported cstate change: term changed from 0 to 1, leader changed from <none> to 43df296ac3e940ea936e8d31e95817be (127.28.210.66). New cstate: current_term: 1 leader_uuid: "43df296ac3e940ea936e8d31e95817be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } health_report { overall_health: UNKNOWN } } }
Aug 06 12:23:24 dist-test-slave-lgr5 krb5kdc[31180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:24.542918 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.530406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.28.210.126:40931)
0806 12:23:24.530580 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.530586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.530979 (+ 393us) server_negotiation.cc:407] Connection header received
0806 12:23:24.531126 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.531130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.531198 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.531341 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:24.532361 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.533208 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.534354 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.534736 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.534882 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.538454 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:24.538460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:24.538463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:24.538495 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:24.540486 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:24.541230 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:24.541234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:24.541236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:24.541346 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:24.541706 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:24.541710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:24.541711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:24.541959 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:24.542072 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.542494 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.542722 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":192,"server-negotiator.queue_time_us":75}
I20260806 12:23:24.542929 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.530237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40931 (local address 127.0.0.1:33622)
0806 12:23:24.530748 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:24.530762 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:24.531012 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:24.531377 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:24.531380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:24.532012 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:24.532190 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.532200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.533359 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.533362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.534132 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.534146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.534896 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.534900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.535098 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.535917 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:24.535935 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:24.538228 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:24.540652 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:24.540667 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:24.540676 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:24.541113 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:24.541469 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:24.541472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:24.541474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:24.541596 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:24.542093 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:24.542102 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:24.542262 (+ 160us) client_negotiation.cc:769] Sending connection context
0806 12:23:24.542425 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 12:23:24.542642 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":49,"thread_start_us":230,"threads_started":1}
I20260806 12:23:24.549340 31245 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:33622:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 12:23:24.550374 31245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:24.556420 31470 tablet_service.cc:1467] Processing CreateTablet for tablet f8d56e8558f74215899a5d718dd8fd0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=490e242060a246a1b347cb073bd66fdc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 12:23:24.556923 31470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d56e8558f74215899a5d718dd8fd0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:24.556420 31700 tablet_service.cc:1467] Processing CreateTablet for tablet f8d56e8558f74215899a5d718dd8fd0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=490e242060a246a1b347cb073bd66fdc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 12:23:24.559373 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d56e8558f74215899a5d718dd8fd0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:24.559881 32036 tablet_bootstrap.cc:492] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: Bootstrap starting.
I20260806 12:23:24.559801 31909 tablet_service.cc:1467] Processing CreateTablet for tablet f8d56e8558f74215899a5d718dd8fd0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=490e242060a246a1b347cb073bd66fdc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 12:23:24.560314 31909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d56e8558f74215899a5d718dd8fd0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:24.562371 32036 tablet_bootstrap.cc:654] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:24.563179 32044 tablet_bootstrap.cc:492] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be: Bootstrap starting.
I20260806 12:23:24.564517 32038 tablet_bootstrap.cc:492] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318: Bootstrap starting.
I20260806 12:23:24.564695 32044 tablet_bootstrap.cc:654] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:24.566478 32038 tablet_bootstrap.cc:654] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:24.566506 32036 tablet_bootstrap.cc:492] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: No bootstrap required, opened a new log
I20260806 12:23:24.567028 32036 ts_tablet_manager.cc:1397] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 12:23:24.567063 32044 tablet_bootstrap.cc:492] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be: No bootstrap required, opened a new log
I20260806 12:23:24.567186 32044 ts_tablet_manager.cc:1397] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260806 12:23:24.567704 32036 raft_consensus.cc:348] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.567965 32036 raft_consensus.cc:374] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:24.568055 32036 raft_consensus.cc:729] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4896862d4c4f9a977036f46c79bf10, State: Initialized, Role: FOLLOWER
I20260806 12:23:24.568064 32044 raft_consensus.cc:348] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.568257 32044 raft_consensus.cc:374] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:24.568297 32044 raft_consensus.cc:729] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43df296ac3e940ea936e8d31e95817be, State: Initialized, Role: FOLLOWER
I20260806 12:23:24.568444 32044 consensus_queue.cc:260] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.568444 32036 consensus_queue.cc:260] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.568755 32044 ts_tablet_manager.cc:1428] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 12:23:24.569052 32046 tablet_replica.cc:439] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } }
I20260806 12:23:24.569449 32046 tablet_replica.cc:442] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:24.569442 32048 tablet_replica.cc:439] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } }
I20260806 12:23:24.569599 32048 tablet_replica.cc:442] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:24.570719 32036 ts_tablet_manager.cc:1428] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260806 12:23:24.571511 32038 tablet_bootstrap.cc:492] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318: No bootstrap required, opened a new log
I20260806 12:23:24.571635 32038 ts_tablet_manager.cc:1397] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260806 12:23:24.572585 32038 raft_consensus.cc:348] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.572826 32038 raft_consensus.cc:374] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:24.572901 32038 raft_consensus.cc:729] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06898be4f939488fb08e4943b09c3318, State: Initialized, Role: FOLLOWER
I20260806 12:23:24.573123 32038 consensus_queue.cc:260] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.573557 32038 ts_tablet_manager.cc:1428] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260806 12:23:24.573470 32045 tablet_replica.cc:439] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } }
I20260806 12:23:24.573647 32045 tablet_replica.cc:442] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 12:23:24.623579 31587 tablet.cc:2369] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:23:24.646234 32003 tablet.cc:2369] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:23:24.896958 32046 raft_consensus.cc:482] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:24.897157 32046 raft_consensus.cc:504] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.897625 32046 leader_election.cc:288] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43df296ac3e940ea936e8d31e95817be (127.28.210.66:40127), 06898be4f939488fb08e4943b09c3318 (127.28.210.67:38715)
I20260806 12:23:24.898253 31729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d56e8558f74215899a5d718dd8fd0c" candidate_uuid: "be4896862d4c4f9a977036f46c79bf10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43df296ac3e940ea936e8d31e95817be" is_pre_election: true
I20260806 12:23:24.898253 31937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d56e8558f74215899a5d718dd8fd0c" candidate_uuid: "be4896862d4c4f9a977036f46c79bf10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06898be4f939488fb08e4943b09c3318" is_pre_election: true
I20260806 12:23:24.898442 31937 raft_consensus.cc:2454] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be4896862d4c4f9a977036f46c79bf10 in term 0.
I20260806 12:23:24.898460 31729 raft_consensus.cc:2454] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be4896862d4c4f9a977036f46c79bf10 in term 0.
I20260806 12:23:24.898842 31391 leader_election.cc:302] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06898be4f939488fb08e4943b09c3318, 43df296ac3e940ea936e8d31e95817be, be4896862d4c4f9a977036f46c79bf10; no voters:
I20260806 12:23:24.899048 32046 raft_consensus.cc:2781] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:23:24.899117 32046 raft_consensus.cc:482] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:24.899144 32046 raft_consensus.cc:3037] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:24.900279 32046 raft_consensus.cc:504] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.900684 32046 leader_election.cc:288] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [CANDIDATE]: Term 1 election: Requested vote from peers 43df296ac3e940ea936e8d31e95817be (127.28.210.66:40127), 06898be4f939488fb08e4943b09c3318 (127.28.210.67:38715)
I20260806 12:23:24.901142 31729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d56e8558f74215899a5d718dd8fd0c" candidate_uuid: "be4896862d4c4f9a977036f46c79bf10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43df296ac3e940ea936e8d31e95817be"
I20260806 12:23:24.901278 31729 raft_consensus.cc:3037] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:24.901250 31937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d56e8558f74215899a5d718dd8fd0c" candidate_uuid: "be4896862d4c4f9a977036f46c79bf10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06898be4f939488fb08e4943b09c3318"
I20260806 12:23:24.901389 31937 raft_consensus.cc:3037] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:24.902477 31729 raft_consensus.cc:2454] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be4896862d4c4f9a977036f46c79bf10 in term 1.
I20260806 12:23:24.902496 31937 raft_consensus.cc:2454] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be4896862d4c4f9a977036f46c79bf10 in term 1.
I20260806 12:23:24.902829 31391 leader_election.cc:302] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43df296ac3e940ea936e8d31e95817be, be4896862d4c4f9a977036f46c79bf10; no voters:
I20260806 12:23:24.903012 32046 raft_consensus.cc:2781] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:24.903184 32046 raft_consensus.cc:686] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [term 1 LEADER]: Becoming Leader. State: Replica: be4896862d4c4f9a977036f46c79bf10, State: Running, Role: LEADER
I20260806 12:23:24.903331 32046 consensus_queue.cc:237] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } }
I20260806 12:23:24.904112 32100 tablet_replica.cc:439] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: TxnStatusTablet state changed. Reason: New leader be4896862d4c4f9a977036f46c79bf10. Latest consensus state: current_term: 1 leader_uuid: "be4896862d4c4f9a977036f46c79bf10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } }
I20260806 12:23:24.904238 32100 tablet_replica.cc:442] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: This TxnStatusTablet replica's current role is: LEADER
I20260806 12:23:24.904577 32105 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 12:23:24.904783 32105 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 12:23:24.905309 31245 catalog_manager.cc:5495] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 reported cstate change: term changed from 0 to 1, leader changed from <none> to be4896862d4c4f9a977036f46c79bf10 (127.28.210.65). New cstate: current_term: 1 leader_uuid: "be4896862d4c4f9a977036f46c79bf10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } health_report { overall_health: UNKNOWN } } }
I20260806 12:23:24.951795 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.948194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:42559 (local address 127.0.0.1:45446)
0806 12:23:24.948377 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:24.948394 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:24.948542 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:24.948922 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:24.948925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:24.948933 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 12:23:24.949061 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.949066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.949870 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.949874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.950509 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.950518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.951010 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.951014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.951099 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.951155 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 12:23:24.951470 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 12:23:24.951486 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 12:23:24.951546 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 12:23:24.951646 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 12:23:24.951795 32031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.948295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.28.210.65:42559)
0806 12:23:24.948415 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.948420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.948440 (+ 20us) server_negotiation.cc:407] Connection header received
0806 12:23:24.948740 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.948743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.948800 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.948871 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:24.949192 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.949766 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.950663 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.950884 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.950977 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.951300 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:24.951385 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:24.951444 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.951571 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.951633 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 12:23:24.955134 31729 raft_consensus.cc:1261] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be [term 1 FOLLOWER]: Refusing update from remote peer be4896862d4c4f9a977036f46c79bf10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:24.955143 31937 raft_consensus.cc:1261] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318 [term 1 FOLLOWER]: Refusing update from remote peer be4896862d4c4f9a977036f46c79bf10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:24.955586 32046 consensus_queue.cc:1035] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:24.955741 32100 consensus_queue.cc:1035] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:24.957516 32048 tablet_replica.cc:439] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be: TxnStatusTablet state changed. Reason: New leader be4896862d4c4f9a977036f46c79bf10. Latest consensus state: current_term: 1 leader_uuid: "be4896862d4c4f9a977036f46c79bf10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } }
I20260806 12:23:24.957640 32048 tablet_replica.cc:442] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:24.958580 32045 tablet_replica.cc:439] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318: TxnStatusTablet state changed. Reason: New leader be4896862d4c4f9a977036f46c79bf10. Latest consensus state: current_term: 1 leader_uuid: "be4896862d4c4f9a977036f46c79bf10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } }
I20260806 12:23:24.958715 32045 tablet_replica.cc:442] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:24.961440 32046 tablet_replica.cc:439] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be4896862d4c4f9a977036f46c79bf10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } }
I20260806 12:23:24.961592 32046 tablet_replica.cc:442] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: This TxnStatusTablet replica's current role is: LEADER
I20260806 12:23:24.961722 32046 tablet_replica.cc:439] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be4896862d4c4f9a977036f46c79bf10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } }
I20260806 12:23:24.961795 32046 tablet_replica.cc:442] T f8d56e8558f74215899a5d718dd8fd0c P be4896862d4c4f9a977036f46c79bf10: This TxnStatusTablet replica's current role is: LEADER
I20260806 12:23:24.962093 32048 tablet_replica.cc:439] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be4896862d4c4f9a977036f46c79bf10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } }
I20260806 12:23:24.962212 32048 tablet_replica.cc:442] T f8d56e8558f74215899a5d718dd8fd0c P 43df296ac3e940ea936e8d31e95817be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:24.962424 32045 tablet_replica.cc:439] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be4896862d4c4f9a977036f46c79bf10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 } } peers { permanent_uuid: "43df296ac3e940ea936e8d31e95817be" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 40127 } } peers { permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 } } }
I20260806 12:23:24.962545 32045 tablet_replica.cc:442] T f8d56e8558f74215899a5d718dd8fd0c P 06898be4f939488fb08e4943b09c3318: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:24.964574 32115 mvcc.cc:205] Tried to move back new op lower bound from 7315533844290822144 to 7315533844085817344. Current Snapshot: MvccSnapshot[applied={T|T < 7315533844290822144}]
I20260806 12:23:24.979000 32128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.973370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.28.210.66:40127)
0806 12:23:24.973752 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.973758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.973851 (+ 93us) server_negotiation.cc:407] Connection header received
0806 12:23:24.973965 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.973969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.974065 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.974343 (+ 278us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:24.974610 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.975468 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.976523 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.976902 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.977034 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.977551 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:24.978263 (+ 712us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:24.978397 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.978717 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.978799 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":165,"threads_started":1}
I20260806 12:23:24.983371 32048 consensus_queue.cc:1035] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [LEADER]: Connected to new peer: Peer: permanent_uuid: "06898be4f939488fb08e4943b09c3318" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:24.989014 32048 consensus_queue.cc:1035] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [LEADER]: Connected to new peer: Peer: permanent_uuid: "be4896862d4c4f9a977036f46c79bf10" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 42559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:24.996708 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.993152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:42559 (local address 127.28.210.66:56179)
0806 12:23:24.993501 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:24.993517 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:24.993631 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:24.993850 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:24.993852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:24.993860 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 12:23:24.993961 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.993966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.994795 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.994799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.995434 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:24.995443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.995888 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.995891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:24.995989 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.996060 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 12:23:24.996394 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 12:23:24.996403 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 12:23:24.996464 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 12:23:24.996568 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":113,"threads_started":1}
I20260806 12:23:24.996704 32031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:24.993214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:56179 (local address 127.28.210.65:42559)
0806 12:23:24.993484 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:24.993488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:24.993584 (+ 96us) server_negotiation.cc:407] Connection header received
0806 12:23:24.993720 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:24.993723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:24.993772 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:24.993832 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:24.994060 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.994688 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.995558 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:24.995786 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:24.995872 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:24.996232 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:24.996318 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:24.996379 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:24.996473 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 12:23:24.996539 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260806 12:23:25.033602 32128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:25.028526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:58971 (local address 127.28.210.66:40127)
0806 12:23:25.028655 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:25.028660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:25.029034 (+ 374us) server_negotiation.cc:407] Connection header received
0806 12:23:25.029367 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:25.029371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:25.029443 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:25.029507 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:25.030033 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:25.030756 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:25.032046 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:25.032369 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:25.032482 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:25.032939 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:25.033054 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:25.033132 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:25.033341 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 12:23:25.033419 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 12:23:25.034363 32138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:25.028278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:40127 (local address 127.28.210.65:58971)
0806 12:23:25.028813 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:25.028833 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:25.029178 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:25.029696 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:25.029699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:25.029709 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 12:23:25.029883 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:25.029892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:25.030889 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:25.030893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:25.031868 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:25.031885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:25.032517 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:25.032522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:25.032630 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:25.032715 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 12:23:25.033165 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 12:23:25.033175 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 12:23:25.034047 (+ 872us) client_negotiation.cc:240] Negotiation successful
0806 12:23:25.034169 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":116,"threads_started":1}
I20260806 12:23:25.074332 31247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:25.074604 31247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:25.078310 31247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c91efe66857e49e9af049e80da1b5017: Sending DeleteTablet for 3 replicas of tablet 5b4d758afa3c4bbab61d8ab0846678b0
I20260806 12:23:25.079448 31909 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4d758afa3c4bbab61d8ab0846678b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:25 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:53792
I20260806 12:23:25.079828 31700 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4d758afa3c4bbab61d8ab0846678b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:25 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:49424
I20260806 12:23:25.080595 32142 tablet_replica.cc:330] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be: stopping tablet replica
I20260806 12:23:25.080823 32142 raft_consensus.cc:2229] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:23:25.081282 32142 raft_consensus.cc:2258] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:25.081753 31470 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4d758afa3c4bbab61d8ab0846678b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:25 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:45434
I20260806 12:23:25.082295 32141 tablet_replica.cc:330] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318: stopping tablet replica
I20260806 12:23:25.082471 32141 raft_consensus.cc:2229] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:25.082653 32141 raft_consensus.cc:2258] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:25.082830 32142 ts_tablet_manager.cc:1905] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:25.083633 32141 ts_tablet_manager.cc:1905] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:25.088120 32141 ts_tablet_manager.cc:1918] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:23:25.088253 32141 log.cc:1196] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/5b4d758afa3c4bbab61d8ab0846678b0
I20260806 12:23:25.088580 32141 ts_tablet_manager.cc:1939] T 5b4d758afa3c4bbab61d8ab0846678b0 P 06898be4f939488fb08e4943b09c3318: Deleting consensus metadata
I20260806 12:23:25.088850 32142 ts_tablet_manager.cc:1918] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:23:25.088975 32142 log.cc:1196] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/5b4d758afa3c4bbab61d8ab0846678b0
I20260806 12:23:25.089262 32142 ts_tablet_manager.cc:1939] T 5b4d758afa3c4bbab61d8ab0846678b0 P 43df296ac3e940ea936e8d31e95817be: Deleting consensus metadata
I20260806 12:23:25.089543 31222 catalog_manager.cc:4841] TS 06898be4f939488fb08e4943b09c3318 (127.28.210.67:38715): tablet 5b4d758afa3c4bbab61d8ab0846678b0 (table test-table [id=dfad2d7f578948b4a8e698661cc06481]) successfully deleted
I20260806 12:23:25.090215 31223 catalog_manager.cc:4841] TS 43df296ac3e940ea936e8d31e95817be (127.28.210.66:40127): tablet 5b4d758afa3c4bbab61d8ab0846678b0 (table test-table [id=dfad2d7f578948b4a8e698661cc06481]) successfully deleted
I20260806 12:23:25.092290 32145 tablet_replica.cc:330] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10: stopping tablet replica
I20260806 12:23:25.092480 32145 raft_consensus.cc:2229] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:25.092669 32145 raft_consensus.cc:2258] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:25.093484 32145 ts_tablet_manager.cc:1905] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:25.098668 32145 ts_tablet_manager.cc:1918] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:23:25.098862 32145 log.cc:1196] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786018995816160-29513-0/minicluster-data/ts-0/wal/wals/5b4d758afa3c4bbab61d8ab0846678b0
I20260806 12:23:25.099258 32145 ts_tablet_manager.cc:1939] T 5b4d758afa3c4bbab61d8ab0846678b0 P be4896862d4c4f9a977036f46c79bf10: Deleting consensus metadata
Aug 06 12:23:25 dist-test-slave-lgr5 krb5kdc[31180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.210.65@KRBTEST.COM
I20260806 12:23:25.100869 31223 catalog_manager.cc:4841] TS be4896862d4c4f9a977036f46c79bf10 (127.28.210.65:42559): tablet 5b4d758afa3c4bbab61d8ab0846678b0 (table test-table [id=dfad2d7f578948b4a8e698661cc06481]) successfully deleted
I20260806 12:23:25.107029 32031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:25.089604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.28.210.65:42559)
0806 12:23:25.089859 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:25.089865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:25.090490 (+ 625us) server_negotiation.cc:407] Connection header received
0806 12:23:25.090666 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:25.090670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:25.090732 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:25.090845 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:25.091627 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:25.092509 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:25.093884 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:25.094213 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:25.094344 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:25.100824 (+ 6480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:25.100845 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:25.100873 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:25.100921 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:25.103922 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:25.104803 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:25.104812 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:25.104815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:25.104955 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:25.105433 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:25.105438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:25.105440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:25.105942 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:25.106276 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:25.106530 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 12:23:25.106788 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
W20260806 12:23:25.108032 31572 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45460
I20260806 12:23:25.120062 32149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:25.110397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.28.210.126:40931)
0806 12:23:25.110753 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:25.110759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:25.110923 (+ 164us) server_negotiation.cc:407] Connection header received
0806 12:23:25.111083 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:25.111088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:25.111149 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:25.111242 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:25.112031 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:25.112700 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:25.113663 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:25.113966 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:25.114078 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:25.115794 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:25.115799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:25.115802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:25.115837 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:25.118013 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:25.118564 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:25.118569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:25.118571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:25.118678 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:25.118961 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:25.118965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:25.118967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:25.119220 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:25.119364 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:25.119695 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 12:23:25.119860 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":94,"threads_started":1}
W20260806 12:23:25.120720 31247 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33634
I20260806 12:23:25.122718 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 31363
I20260806 12:23:25.146239 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 31598
I20260806 12:23:25.157729 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 31816
I20260806 12:23:25.178941 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 31199
2026-08-06T12:23:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3499 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:23:25 dist-test-slave-lgr5 krb5kdc[32153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:23:25 dist-test-slave-lgr5 krb5kdc[32153](info): set up 2 sockets
Aug 06 12:23:25 dist-test-slave-lgr5 krb5kdc[32153](info): commencing operation
krb5kdc: starting...
W20260806 12:23:26.403486 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:23:26 dist-test-slave-lgr5 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:23:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:23:26.552356 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:39895
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:39895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:26.685622 32171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:26.685986 32171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:26.686044 32171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:26.690547 32171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:26.690626 32171 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 12:23:26.690644 32171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:26.690660 32171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:26.690676 32171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:26.690691 32171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:26.696664 32171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:39895
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:39895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:26.697921 32171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:26.699040 32171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:23:26.709846 32171 server_base.cc:1034] running on GCE node
W20260806 12:23:26.709789 32180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:26.709772 32181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:26.709767 32183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:26.710852 32171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:26.712361 32171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:26.713557 32171 hybrid_clock.cc:648] HybridClock initialized: now 1786019006713525 us; error 59 us; skew 500 ppm
Aug 06 12:23:26 dist-test-slave-lgr5 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:26.722088 32171 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:23:26.723398 32171 webserver.cc:460] Webserver started at http://127.28.210.126:43297/ using document root <none> and password file <none>
I20260806 12:23:26.724210 32171 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:26.724308 32171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:26.724560 32171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:26.727140 32171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "36d7c70870da48f2a1956a6c961e5be0"
format_stamp: "Formatted at 2026-08-06 12:23:26 on dist-test-slave-lgr5"
server_key: "de85fc5b830e9d0fa9ca393cb43434bc"
server_key_iv: "2e26813e5c13833ec7942bfa7cc8dd85"
server_key_version: "encryptionkey@0"
I20260806 12:23:26.727898 32171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "36d7c70870da48f2a1956a6c961e5be0"
format_stamp: "Formatted at 2026-08-06 12:23:26 on dist-test-slave-lgr5"
server_key: "de85fc5b830e9d0fa9ca393cb43434bc"
server_key_iv: "2e26813e5c13833ec7942bfa7cc8dd85"
server_key_version: "encryptionkey@0"
I20260806 12:23:26.732522 32171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 12:23:26.735870 32192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:26.737620 32171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:26.737835 32171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "36d7c70870da48f2a1956a6c961e5be0"
format_stamp: "Formatted at 2026-08-06 12:23:26 on dist-test-slave-lgr5"
server_key: "de85fc5b830e9d0fa9ca393cb43434bc"
server_key_iv: "2e26813e5c13833ec7942bfa7cc8dd85"
server_key_version: "encryptionkey@0"
I20260806 12:23:26.738109 32171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:26.758662 32171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:26.762358 32171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:26.762588 32171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:26.771332 32171 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:39895
I20260806 12:23:26.771349 32286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:39895 every 8 connection(s)
I20260806 12:23:26.772526 32171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:26.776190 32287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:26.781935 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 32171
I20260806 12:23:26.782063 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:26.782370 29513 external_mini_cluster.cc:1428] Setting key f4afd671a924b72583e013169e1e1e96
I20260806 12:23:26.783521 32287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0: Bootstrap starting.
I20260806 12:23:26.786969 32287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:26.788302 32287 log.cc:824] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:26.791180 32287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0: No bootstrap required, opened a new log
Aug 06 12:23:26 dist-test-slave-lgr5 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:26.796375 32287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d7c70870da48f2a1956a6c961e5be0" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 39895 } }
I20260806 12:23:26.796787 32287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:26.796869 32287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d7c70870da48f2a1956a6c961e5be0, State: Initialized, Role: FOLLOWER
I20260806 12:23:26.797475 32287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d7c70870da48f2a1956a6c961e5be0" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 39895 } }
I20260806 12:23:26.797655 32287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:26.797736 32287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:26.797860 32287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:26.798949 32287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d7c70870da48f2a1956a6c961e5be0" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 39895 } }
I20260806 12:23:26.799386 32287 leader_election.cc:302] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36d7c70870da48f2a1956a6c961e5be0; no voters:
I20260806 12:23:26.799896 32287 leader_election.cc:288] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:26.800004 32296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:26.800287 32296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [term 1 LEADER]: Becoming Leader. State: Replica: 36d7c70870da48f2a1956a6c961e5be0, State: Running, Role: LEADER
I20260806 12:23:26.800664 32296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d7c70870da48f2a1956a6c961e5be0" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 39895 } }
I20260806 12:23:26.800887 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:26.784384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.28.210.126:39895)
0806 12:23:26.784979 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:26.785002 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:26.785078 (+ 76us) server_negotiation.cc:407] Connection header received
0806 12:23:26.786149 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:26.786183 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:26.786746 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:26.787249 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:26.788143 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:26.789976 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:26.790993 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:26.791741 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:26.791922 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:26.794765 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:26.794774 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:26.794801 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:26.794852 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:26.797111 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:26.797703 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:26.797711 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:26.797720 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:26.797911 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:26.798232 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:26.798236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:26.798238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:26.798792 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:26.799013 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:26.799390 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 12:23:26.799771 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":170,"threads_started":1}
I20260806 12:23:26.801376 32287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:26.802045 32296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36d7c70870da48f2a1956a6c961e5be0. Latest consensus state: current_term: 1 leader_uuid: "36d7c70870da48f2a1956a6c961e5be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d7c70870da48f2a1956a6c961e5be0" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 39895 } } }
I20260806 12:23:26.802269 32296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:26.804168 32310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:26.804728 32297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36d7c70870da48f2a1956a6c961e5be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d7c70870da48f2a1956a6c961e5be0" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 39895 } } }
I20260806 12:23:26.804827 32297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:26.807056 32310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:26.813686 32310 catalog_manager.cc:1347] Generated new cluster ID: 5d92bd4ab5644ff79811ef34004d5ac7
I20260806 12:23:26.813786 32310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:26.834849 32310 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:26.836169 32310 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:26.851747 32310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0: Generated new TSK 0
I20260806 12:23:26.852716 32310 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:23:26.892206 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:39895
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:27.030779 32327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:27.031114 32327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:27.031167 32327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:27.035435 32327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:27.035526 32327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:27.035569 32327 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 12:23:27.035583 32327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 12:23:27.035615 32327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:23:27.041625 32327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.210.126:39895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:27.042901 32327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:27.044286 32327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:27.056898 32337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:27.056964 32334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:27.057030 32333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:27.057281 32327 server_base.cc:1034] running on GCE node
I20260806 12:23:27.057847 32327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:27.058594 32327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:27.059880 32327 hybrid_clock.cc:648] HybridClock initialized: now 1786019007059841 us; error 55 us; skew 500 ppm
Aug 06 12:23:27 dist-test-slave-lgr5 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:27.063366 32327 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:23:27.064520 32327 webserver.cc:460] Webserver started at http://127.28.210.65:45735/ using document root <none> and password file <none>
I20260806 12:23:27.065188 32327 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:27.065267 32327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:27.065521 32327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:27.067974 32327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "1a718cad601d4fda828474f2d5249b1f"
format_stamp: "Formatted at 2026-08-06 12:23:27 on dist-test-slave-lgr5"
server_key: "cc6deaa25b7ce973a9ed2187c63d661a"
server_key_iv: "3601e3216e19dc85e778bd5e747941ee"
server_key_version: "encryptionkey@0"
I20260806 12:23:27.068734 32327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "1a718cad601d4fda828474f2d5249b1f"
format_stamp: "Formatted at 2026-08-06 12:23:27 on dist-test-slave-lgr5"
server_key: "cc6deaa25b7ce973a9ed2187c63d661a"
server_key_iv: "3601e3216e19dc85e778bd5e747941ee"
server_key_version: "encryptionkey@0"
I20260806 12:23:27.073402 32327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:23:27.077991 32350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:27.079620 32327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 12:23:27.079793 32327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "1a718cad601d4fda828474f2d5249b1f"
format_stamp: "Formatted at 2026-08-06 12:23:27 on dist-test-slave-lgr5"
server_key: "cc6deaa25b7ce973a9ed2187c63d661a"
server_key_iv: "3601e3216e19dc85e778bd5e747941ee"
server_key_version: "encryptionkey@0"
I20260806 12:23:27.079950 32327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:27.109134 32327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:27.112996 32327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:27.113214 32327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:27.115607 32327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:27.115674 32327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:27.115720 32327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:27.115736 32327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 12:23:27 dist-test-slave-lgr5 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:27.129698 32327 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:33339
I20260806 12:23:27.129750 32538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:33339 every 8 connection(s)
I20260806 12:23:27.130934 32327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:23:27.132272 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.116564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:50185 (local address 127.28.210.126:39895)
0806 12:23:27.116734 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.116739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.117843 (+ 1104us) server_negotiation.cc:407] Connection header received
0806 12:23:27.119238 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.119243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.119312 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.119373 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:27.121606 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.122365 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.123992 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.124366 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.124487 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.127889 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:27.127894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:27.127896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:27.127938 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:27.129618 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.130335 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.130339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.130340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.130433 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.130873 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.130876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.130877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.131145 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:27.131243 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.131937 (+ 694us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.132129 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 12:23:27.132722 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 32327
I20260806 12:23:27.132834 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:23:27.133102 29513 external_mini_cluster.cc:1428] Setting key e647c0887156c35983c70badec174c30
I20260806 12:23:27.133397 32360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.116998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:39895 (local address 127.28.210.65:50185)
0806 12:23:27.117619 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:27.117693 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:27.118916 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:27.119883 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:27.119893 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:27.120493 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:27.121322 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.121347 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.122579 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.122585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.123796 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.123816 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.124601 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.124606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.124838 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.125764 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:27.125807 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:27.127560 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:27.129781 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.129788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.129805 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.130189 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.130540 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.130543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.130545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.130736 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.131268 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:27.131276 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:27.131652 (+ 376us) client_negotiation.cc:769] Sending connection context
0806 12:23:27.131865 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 12:23:27.132273 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":111,"threads_started":1}
I20260806 12:23:27.135036 32539 heartbeater.cc:347] Connected to a master server at 127.28.210.126:39895
I20260806 12:23:27.135210 32539 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:27.135848 32539 heartbeater.cc:511] Master 127.28.210.126:39895 requested a full tablet report, sending...
I20260806 12:23:27.137734 32227 ts_manager.cc:194] Registered new tserver with Master: 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339)
I20260806 12:23:27.139353 32227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:50185
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:23:27.146405 32550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.137716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:39895 (local address 127.28.210.65:50389)
0806 12:23:27.138163 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:27.138179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:27.138301 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:27.138644 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:27.138672 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:27.138933 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:27.139074 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.139082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.140252 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.140256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.140948 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.140959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.141485 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.141488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.141585 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.142143 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:27.142156 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:27.142691 (+ 535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:27.144657 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.144661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.144664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.144881 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.145217 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.145220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.145222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.145315 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.145760 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:27.145763 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:27.145906 (+ 143us) client_negotiation.cc:769] Sending connection context
0806 12:23:27.146005 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 12:23:27.146184 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":250,"threads_started":1}
I20260806 12:23:27.146401 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.137802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:50389 (local address 127.28.210.126:39895)
0806 12:23:27.137945 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.137950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.138270 (+ 320us) server_negotiation.cc:407] Connection header received
0806 12:23:27.138425 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.138429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.138488 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.138568 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:27.139209 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.139975 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.141094 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.141357 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.141460 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.142843 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:27.142847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:27.142850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:27.142879 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:27.144531 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.145004 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.145015 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.145017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.145121 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.145423 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.145426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.145428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.145651 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:27.145754 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.146093 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.146224 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:23:27.168197 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:39895
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:27.290894 32553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:27.291213 32553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:27.291265 32553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:27.295745 32553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:27.295872 32553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:27.295930 32553 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 12:23:27.295962 32553 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 12:23:27.296016 32553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:23:27.302330 32553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.210.126:39895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:27.303938 32553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:27.305544 32553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:27.318305 32564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:27.318353 32565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:27.318305 32567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:27.318593 32553 server_base.cc:1034] running on GCE node
I20260806 12:23:27.319118 32553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:27.319887 32553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:27.321067 32553 hybrid_clock.cc:648] HybridClock initialized: now 1786019007321042 us; error 39 us; skew 500 ppm
Aug 06 12:23:27 dist-test-slave-lgr5 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:27.324045 32553 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:23:27.325143 32553 webserver.cc:460] Webserver started at http://127.28.210.66:42745/ using document root <none> and password file <none>
I20260806 12:23:27.325770 32553 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:27.325845 32553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:27.326145 32553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:27.328579 32553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "21523712491c4b759bc7e48ee84123e1"
format_stamp: "Formatted at 2026-08-06 12:23:27 on dist-test-slave-lgr5"
server_key: "0c5e177af4ce32537ac4c73a79fa7ff4"
server_key_iv: "4b8d5967b84f9e5737b2a89811f343e4"
server_key_version: "encryptionkey@0"
I20260806 12:23:27.329216 32553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "21523712491c4b759bc7e48ee84123e1"
format_stamp: "Formatted at 2026-08-06 12:23:27 on dist-test-slave-lgr5"
server_key: "0c5e177af4ce32537ac4c73a79fa7ff4"
server_key_iv: "4b8d5967b84f9e5737b2a89811f343e4"
server_key_version: "encryptionkey@0"
I20260806 12:23:27.333518 32553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:23:27.336460 32581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:27.337911 32553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 12:23:27.338047 32553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "21523712491c4b759bc7e48ee84123e1"
format_stamp: "Formatted at 2026-08-06 12:23:27 on dist-test-slave-lgr5"
server_key: "0c5e177af4ce32537ac4c73a79fa7ff4"
server_key_iv: "4b8d5967b84f9e5737b2a89811f343e4"
server_key_version: "encryptionkey@0"
I20260806 12:23:27.338173 32553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:27.351291 32553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:27.355527 32553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:27.355813 32553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:27.357614 32553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:27.357758 32553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:27.357873 32553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:27.357908 32553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 12:23:27 dist-test-slave-lgr5 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:27.370771 32553 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:35587
I20260806 12:23:27.370810 320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:35587 every 8 connection(s)
I20260806 12:23:27.372047 32553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:23:27.375267 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.358946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:56401 (local address 127.28.210.126:39895)
0806 12:23:27.359094 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.359098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.360618 (+ 1520us) server_negotiation.cc:407] Connection header received
0806 12:23:27.362032 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.362036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.362108 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.362206 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:27.364234 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.365047 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.367022 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.367395 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.367510 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.371082 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:27.371087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:27.371090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:27.371127 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:27.372713 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.373419 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.373423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.373424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.373522 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.373891 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.373895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.373897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.374115 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:27.374218 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.374795 (+ 577us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.375078 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 12:23:27.376870 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.359514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:39895 (local address 127.28.210.66:56401)
0806 12:23:27.360407 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:27.360487 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:27.361719 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:27.362503 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:27.362514 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:27.363130 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:27.363985 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.364013 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.365234 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.365237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.366846 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.366864 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.367653 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.367663 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.367906 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.368756 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:27.368782 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:27.370787 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:27.372899 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.372906 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.372922 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.373290 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.373621 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.373625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.373627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.373782 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.374236 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:27.374244 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:27.374529 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 12:23:27.374711 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 12:23:27.375852 (+ 1141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":165,"threads_started":1}
I20260806 12:23:27.378134 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 32553
I20260806 12:23:27.378392 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:23:27.379050 29513 external_mini_cluster.cc:1428] Setting key 26743d50dee4187950eeed1053d055de
I20260806 12:23:27.379588 322 heartbeater.cc:347] Connected to a master server at 127.28.210.126:39895
I20260806 12:23:27.379773 322 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:27.380496 322 heartbeater.cc:511] Master 127.28.210.126:39895 requested a full tablet report, sending...
I20260806 12:23:27.381906 32226 ts_manager.cc:194] Registered new tserver with Master: 21523712491c4b759bc7e48ee84123e1 (127.28.210.66:35587)
I20260806 12:23:27.382727 32226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:56401
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
I20260806 12:23:27.392549 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.382585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:39895 (local address 127.28.210.66:48911)
0806 12:23:27.383157 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:27.383179 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:27.383321 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:27.384579 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:27.384582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:27.384839 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:27.384976 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.384985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.386121 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.386124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.386817 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.386831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.387420 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.387423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.387601 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.388182 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:27.388195 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:27.388557 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:27.390608 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.390611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.390613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.390829 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.391139 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.391142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.391143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.391225 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.391885 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:27.391888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:27.391988 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 12:23:27.392124 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 12:23:27.392356 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":133,"threads_started":1}
I20260806 12:23:27.392696 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.382748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:48911 (local address 127.28.210.126:39895)
0806 12:23:27.382939 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.382945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.383238 (+ 293us) server_negotiation.cc:407] Connection header received
0806 12:23:27.384268 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.384272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.384332 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.384402 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:27.385138 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.385965 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.386975 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.387277 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.387414 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.388728 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:27.388733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:27.388736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:27.388769 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:27.390471 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.390935 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.390939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.390941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.391037 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.391320 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.391324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.391325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.391696 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:27.391827 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.392331 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.392485 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:23:27.416327 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:39895
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:27.540128 335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:27.540493 335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:27.540565 335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:27.545518 335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:27.545619 335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:27.545682 335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 12:23:27.545710 335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 12:23:27.545769 335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:23:27.552510 335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.210.126:39895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:27.553952 335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:27.555322 335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:27.568028 349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:27.568028 347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:27.568018 343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:27.568608 335 server_base.cc:1034] running on GCE node
I20260806 12:23:27.569206 335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:27.570114 335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:27.571329 335 hybrid_clock.cc:648] HybridClock initialized: now 1786019007571293 us; error 53 us; skew 500 ppm
Aug 06 12:23:27 dist-test-slave-lgr5 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:27.574777 335 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:23:27.575932 335 webserver.cc:460] Webserver started at http://127.28.210.67:44851/ using document root <none> and password file <none>
I20260806 12:23:27.576632 335 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:27.576709 335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:27.576941 335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:27.579350 335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "9f9eafc562da4dfea97fe03db90f27fe"
format_stamp: "Formatted at 2026-08-06 12:23:27 on dist-test-slave-lgr5"
server_key: "349c7309a2ad0f431087a50cb9bde3d2"
server_key_iv: "e279d6b60c375cf179d17ab99c06cfd0"
server_key_version: "encryptionkey@0"
I20260806 12:23:27.580039 335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "9f9eafc562da4dfea97fe03db90f27fe"
format_stamp: "Formatted at 2026-08-06 12:23:27 on dist-test-slave-lgr5"
server_key: "349c7309a2ad0f431087a50cb9bde3d2"
server_key_iv: "e279d6b60c375cf179d17ab99c06cfd0"
server_key_version: "encryptionkey@0"
I20260806 12:23:27.585144 335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260806 12:23:27.588848 359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:27.590541 335 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:23:27.590718 335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "9f9eafc562da4dfea97fe03db90f27fe"
format_stamp: "Formatted at 2026-08-06 12:23:27 on dist-test-slave-lgr5"
server_key: "349c7309a2ad0f431087a50cb9bde3d2"
server_key_iv: "e279d6b60c375cf179d17ab99c06cfd0"
server_key_version: "encryptionkey@0"
I20260806 12:23:27.590869 335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:27.606575 335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:27.610550 335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:27.610824 335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:27.612751 335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:27.612830 335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:27.612897 335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:27.612926 335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 12:23:27 dist-test-slave-lgr5 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:27.625131 335 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:41935
I20260806 12:23:27.625154 539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:41935 every 8 connection(s)
I20260806 12:23:27.626425 335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:23:27.630746 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.614063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:38447 (local address 127.28.210.126:39895)
0806 12:23:27.614222 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.614226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.615799 (+ 1573us) server_negotiation.cc:407] Connection header received
0806 12:23:27.617264 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.617268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.617347 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.617455 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:27.619471 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.620365 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.621898 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.622258 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.622396 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.625841 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:27.625845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:27.625848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:27.625900 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:27.627983 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.628797 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.628801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.628803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.628919 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.629419 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.629423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.629424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.629703 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:27.629808 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.630405 (+ 597us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.630565 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 12:23:27.631700 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.614730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:39895 (local address 127.28.210.67:38447)
0806 12:23:27.615554 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:27.615636 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:27.616962 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:27.617875 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:27.617897 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:27.618577 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:27.619217 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.619242 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.620547 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.620551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.621742 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.621755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.622468 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.622471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.622702 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.623452 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:27.623474 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:27.625460 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:27.628201 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.628209 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.628228 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.628651 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.629044 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.629048 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.629050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.629209 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.629830 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:27.629838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:27.630141 (+ 303us) client_negotiation.cc:769] Sending connection context
0806 12:23:27.630333 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 12:23:27.630735 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":226,"threads_started":1}
I20260806 12:23:27.633391 540 heartbeater.cc:347] Connected to a master server at 127.28.210.126:39895
I20260806 12:23:27.633589 540 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:27.634231 540 heartbeater.cc:511] Master 127.28.210.126:39895 requested a full tablet report, sending...
I20260806 12:23:27.635640 32227 ts_manager.cc:194] Registered new tserver with Master: 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67:41935)
I20260806 12:23:27.636294 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 335
I20260806 12:23:27.636428 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:23:27.636580 32227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:38447
I20260806 12:23:27.636826 29513 external_mini_cluster.cc:1428] Setting key 1eb65923888725693aad8f269397c9f8
I20260806 12:23:27.639456 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:23:27.645998 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.636568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:51845 (local address 127.28.210.126:39895)
0806 12:23:27.637040 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.637046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.637346 (+ 300us) server_negotiation.cc:407] Connection header received
0806 12:23:27.637464 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.637467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.637524 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.637605 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:27.638378 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.639131 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.640315 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.640574 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.640676 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.642065 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:27.642070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:27.642073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:27.642105 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:27.643797 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.644341 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.644344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.644345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.644454 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.644841 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.644846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.644848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.645082 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:27.645197 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.645531 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.645663 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367}
I20260806 12:23:27.646035 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.636721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:39895 (local address 127.28.210.67:51845)
0806 12:23:27.637247 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:27.637263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:27.637380 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:27.637772 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:27.637775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:27.638054 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:27.638256 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.638266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.639268 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.639272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.640162 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.640178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.640718 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.640722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.640824 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.641389 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:27.641404 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:27.641879 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:27.643933 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.643937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.643940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.644237 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.644580 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.644584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.644587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.644699 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.645210 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:27.645213 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:27.645333 (+ 120us) client_negotiation.cc:769] Sending connection context
0806 12:23:27.645448 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 12:23:27.645656 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":131,"threads_started":1}
Aug 06 12:23:27 dist-test-slave-lgr5 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 12:23:27 dist-test-slave-lgr5 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:27.671263 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.662246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.28.210.126:39895)
0806 12:23:27.662433 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.662467 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.662483 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:23:27.662534 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.662537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.662582 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.662666 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:27.663462 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.664032 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.664982 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.665249 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.665384 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.667980 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:27.667984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:27.667986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:27.668020 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:27.669556 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.670043 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.670046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.670048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.670132 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.670343 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.670345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.670346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.670556 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:27.670659 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.671003 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.671125 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 12:23:27.674881 32227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:23:27.678287 32227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:27.694743 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.689542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38030 (local address 127.28.210.65:33339)
0806 12:23:27.690104 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.690109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.690127 (+ 18us) server_negotiation.cc:407] Connection header received
0806 12:23:27.690189 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.690194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.690335 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.690460 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:27.690649 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.691444 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.693395 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.693958 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.694059 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.694231 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.694472 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.694589 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":101,"threads_started":1}
I20260806 12:23:27.694749 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.688443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:41935 (local address 127.0.0.1:44526)
0806 12:23:27.689478 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:27.689492 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:27.689590 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:27.690128 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:27.690134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:27.690156 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:27.690292 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.690299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.691621 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.691624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.693238 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.693252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.694230 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.694234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.694332 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.694356 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 12:23:27.694413 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 12:23:27.694571 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":123,"thread_start_us":115,"threads_started":1}
I20260806 12:23:27.695137 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.688530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55078 (local address 127.28.210.66:35587)
0806 12:23:27.689990 (+ 1460us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.690002 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.690025 (+ 23us) server_negotiation.cc:407] Connection header received
0806 12:23:27.690101 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.690106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.690299 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.690455 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:27.690874 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.691914 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.693199 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.694023 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.694132 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.694318 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.694858 (+ 540us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.694956 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1353,"thread_start_us":151,"threads_started":1}
I20260806 12:23:27.694777 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.688364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:35587 (local address 127.0.0.1:55078)
0806 12:23:27.688842 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:27.688906 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:27.689054 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:27.690604 (+ 1550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:27.690608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:27.690628 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:27.690738 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.690745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.692036 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.692039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.693073 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.693084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.694392 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.694395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.694483 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.694503 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 12:23:27.694609 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 12:23:27.694675 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":98,"threads_started":1}
I20260806 12:23:27.695384 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.689217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.28.210.67:41935)
0806 12:23:27.689673 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.689685 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.689710 (+ 25us) server_negotiation.cc:407] Connection header received
0806 12:23:27.689789 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.689795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.689967 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.690122 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:27.690403 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.691524 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.693369 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.694104 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.694836 (+ 732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.695005 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.695069 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.695155 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":147,"threads_started":1}
I20260806 12:23:27.694746 563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.689143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:33339 (local address 127.0.0.1:38030)
0806 12:23:27.689757 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:27.689769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:27.689887 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:27.690440 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:27.690443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:27.690464 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:27.690554 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.690560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.691563 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.691567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.693238 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.693252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.694088 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.694092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.694238 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.694294 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 12:23:27.694422 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 12:23:27.694620 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":234,"thread_start_us":96,"threads_started":1}
I20260806 12:23:27.698699 445 tablet_service.cc:1467] Processing CreateTablet for tablet 446fc33a158d436cbbc1aa331996fef8 (DEFAULT_TABLE table=test-table [id=5c681069e71d41109bdcd055274def7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:27.698730 32667 tablet_service.cc:1467] Processing CreateTablet for tablet 446fc33a158d436cbbc1aa331996fef8 (DEFAULT_TABLE table=test-table [id=5c681069e71d41109bdcd055274def7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:27.699791 445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 446fc33a158d436cbbc1aa331996fef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:27.699805 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 446fc33a158d436cbbc1aa331996fef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:27.699968 32428 tablet_service.cc:1467] Processing CreateTablet for tablet 446fc33a158d436cbbc1aa331996fef8 (DEFAULT_TABLE table=test-table [id=5c681069e71d41109bdcd055274def7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:27.701138 32428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 446fc33a158d436cbbc1aa331996fef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:27.707337 570 tablet_bootstrap.cc:492] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe: Bootstrap starting.
I20260806 12:23:27.709085 572 tablet_bootstrap.cc:492] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f: Bootstrap starting.
I20260806 12:23:27.709934 570 tablet_bootstrap.cc:654] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:27.710707 570 log.cc:824] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:27.710958 572 tablet_bootstrap.cc:654] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:27.711766 572 log.cc:824] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:27.711925 571 tablet_bootstrap.cc:492] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1: Bootstrap starting.
I20260806 12:23:27.713084 570 tablet_bootstrap.cc:492] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe: No bootstrap required, opened a new log
I20260806 12:23:27.713308 570 ts_tablet_manager.cc:1397] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 12:23:27.713567 572 tablet_bootstrap.cc:492] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f: No bootstrap required, opened a new log
I20260806 12:23:27.713752 572 ts_tablet_manager.cc:1397] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260806 12:23:27.714581 571 tablet_bootstrap.cc:654] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:27.715507 571 log.cc:824] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:27.717428 570 raft_consensus.cc:348] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } }
I20260806 12:23:27.717756 570 raft_consensus.cc:374] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:27.717819 570 raft_consensus.cc:729] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9eafc562da4dfea97fe03db90f27fe, State: Initialized, Role: FOLLOWER
I20260806 12:23:27.717682 572 raft_consensus.cc:348] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } }
I20260806 12:23:27.717945 572 raft_consensus.cc:374] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:27.718014 572 raft_consensus.cc:729] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a718cad601d4fda828474f2d5249b1f, State: Initialized, Role: FOLLOWER
I20260806 12:23:27.718115 571 tablet_bootstrap.cc:492] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1: No bootstrap required, opened a new log
I20260806 12:23:27.718338 571 ts_tablet_manager.cc:1397] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 12:23:27.718327 570 consensus_queue.cc:260] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } }
I20260806 12:23:27.718721 572 consensus_queue.cc:260] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } }
I20260806 12:23:27.719481 570 ts_tablet_manager.cc:1428] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 12:23:27.719761 572 ts_tablet_manager.cc:1428] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260806 12:23:27.719899 540 heartbeater.cc:503] Master 127.28.210.126:39895 was elected leader, sending a full tablet report...
I20260806 12:23:27.720184 32539 heartbeater.cc:503] Master 127.28.210.126:39895 was elected leader, sending a full tablet report...
I20260806 12:23:27.722337 571 raft_consensus.cc:348] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } }
I20260806 12:23:27.722638 571 raft_consensus.cc:374] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:27.722697 571 raft_consensus.cc:729] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21523712491c4b759bc7e48ee84123e1, State: Initialized, Role: FOLLOWER
I20260806 12:23:27.723171 571 consensus_queue.cc:260] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } }
I20260806 12:23:27.724372 322 heartbeater.cc:503] Master 127.28.210.126:39895 was elected leader, sending a full tablet report...
I20260806 12:23:27.724577 571 ts_tablet_manager.cc:1428] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 12:23:27.860460 576 raft_consensus.cc:482] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:27.860734 576 raft_consensus.cc:504] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } }
I20260806 12:23:27.861974 576 leader_election.cc:288] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21523712491c4b759bc7e48ee84123e1 (127.28.210.66:35587), 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339)
I20260806 12:23:27.866504 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.862376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:40573 (local address 127.28.210.66:35587)
0806 12:23:27.862509 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.862513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.862531 (+ 18us) server_negotiation.cc:407] Connection header received
0806 12:23:27.862700 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.862703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.862756 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.862844 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:27.863125 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.863866 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.865344 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.865972 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.866094 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.866153 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.866295 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.866364 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 12:23:27.866523 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.862253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:35587 (local address 127.28.210.67:40573)
0806 12:23:27.862432 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:27.862451 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:27.862576 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:27.862877 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:27.862880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:27.862903 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:27.863012 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.863017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.864009 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.864013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.865226 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.865236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.866081 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.866084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.866167 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.866201 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 12:23:27.866262 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 12:23:27.866375 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":4}
I20260806 12:23:27.867381 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.862559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:33339 (local address 127.28.210.67:50073)
0806 12:23:27.862999 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:27.863010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:27.863104 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:27.863314 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:27.863317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:27.863335 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:27.864051 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.864062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.865079 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.865081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.866203 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.866213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.867066 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.867070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.867143 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.867162 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 12:23:27.867222 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 12:23:27.867273 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":144,"threads_started":1}
I20260806 12:23:27.867296 32710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "446fc33a158d436cbbc1aa331996fef8" candidate_uuid: "9f9eafc562da4dfea97fe03db90f27fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21523712491c4b759bc7e48ee84123e1" is_pre_election: true
I20260806 12:23:27.867448 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.862694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:50073 (local address 127.28.210.65:33339)
0806 12:23:27.862838 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.862843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.863096 (+ 253us) server_negotiation.cc:407] Connection header received
0806 12:23:27.863187 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.863189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.863232 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.863288 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:27.864210 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.864973 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.866335 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.866924 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.867050 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.867100 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.867256 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.867335 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 12:23:27.867686 32710 raft_consensus.cc:2454] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9eafc562da4dfea97fe03db90f27fe in term 0.
I20260806 12:23:27.868198 32453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "446fc33a158d436cbbc1aa331996fef8" candidate_uuid: "9f9eafc562da4dfea97fe03db90f27fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a718cad601d4fda828474f2d5249b1f" is_pre_election: true
I20260806 12:23:27.868251 360 leader_election.cc:302] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21523712491c4b759bc7e48ee84123e1, 9f9eafc562da4dfea97fe03db90f27fe; no voters:
I20260806 12:23:27.868460 32453 raft_consensus.cc:2454] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9eafc562da4dfea97fe03db90f27fe in term 0.
I20260806 12:23:27.868543 576 raft_consensus.cc:2781] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:23:27.868655 576 raft_consensus.cc:482] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:27.868707 576 raft_consensus.cc:3037] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:27.869851 576 raft_consensus.cc:504] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } }
I20260806 12:23:27.870222 576 leader_election.cc:288] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [CANDIDATE]: Term 1 election: Requested vote from peers 21523712491c4b759bc7e48ee84123e1 (127.28.210.66:35587), 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339)
I20260806 12:23:27.870678 32453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "446fc33a158d436cbbc1aa331996fef8" candidate_uuid: "9f9eafc562da4dfea97fe03db90f27fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a718cad601d4fda828474f2d5249b1f"
I20260806 12:23:27.870689 32710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "446fc33a158d436cbbc1aa331996fef8" candidate_uuid: "9f9eafc562da4dfea97fe03db90f27fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21523712491c4b759bc7e48ee84123e1"
I20260806 12:23:27.870801 32453 raft_consensus.cc:3037] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:27.870841 32710 raft_consensus.cc:3037] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:27.871912 32453 raft_consensus.cc:2454] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9eafc562da4dfea97fe03db90f27fe in term 1.
I20260806 12:23:27.872057 32710 raft_consensus.cc:2454] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9eafc562da4dfea97fe03db90f27fe in term 1.
I20260806 12:23:27.872285 360 leader_election.cc:302] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a718cad601d4fda828474f2d5249b1f, 9f9eafc562da4dfea97fe03db90f27fe; no voters:
I20260806 12:23:27.872452 576 raft_consensus.cc:2781] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:27.872686 576 raft_consensus.cc:686] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 1 LEADER]: Becoming Leader. State: Replica: 9f9eafc562da4dfea97fe03db90f27fe, State: Running, Role: LEADER
I20260806 12:23:27.873049 576 consensus_queue.cc:237] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } }
W20260806 12:23:27.873522 323 tablet.cc:2369] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:23:27.876917 32226 catalog_manager.cc:5495] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67). New cstate: current_term: 1 leader_uuid: "9f9eafc562da4dfea97fe03db90f27fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } health_report { overall_health: UNKNOWN } } }
W20260806 12:23:27.877878 541 tablet.cc:2369] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:23:27.882504 32540 tablet.cc:2369] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 12:23:27 dist-test-slave-lgr5 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:27.964466 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.954367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:39895 (local address 127.0.0.1:46686)
0806 12:23:27.954752 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:27.954767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:27.954884 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:27.955160 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:27.955169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:27.955630 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:27.955738 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.955745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.956679 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.956682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.957394 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:27.957407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.957934 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.957938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:27.958020 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.958738 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:27.958753 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:27.960677 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:27.962438 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.962454 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.962464 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.962825 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.963103 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:27.963106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:27.963108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:27.963200 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:27.963797 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:27.963806 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:27.963942 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 12:23:27.964058 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 12:23:27.964287 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":135,"threads_started":1}
I20260806 12:23:27.964741 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:27.954445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.28.210.126:39895)
0806 12:23:27.954602 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:27.954607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:27.954881 (+ 274us) server_negotiation.cc:407] Connection header received
0806 12:23:27.954954 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:27.954956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:27.955028 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:27.955110 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:27.955933 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.956525 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.957554 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:27.957782 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:27.957876 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:27.960844 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:27.960848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:27.960850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:27.960876 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:27.962300 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.962926 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.962930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.962932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.963015 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:27.963366 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:27.963370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:27.963371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:27.963643 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:27.963750 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:27.964169 (+ 419us) server_negotiation.cc:299] Negotiation successful
0806 12:23:27.964430 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":66}
I20260806 12:23:27.969995 32226 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:46686:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 12:23:27.970943 32226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:27.978158 32667 tablet_service.cc:1467] Processing CreateTablet for tablet aa280d05368c4347b3a1c7596e7f9f16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5600cd7c7db24ef8afd53e0485819738]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 12:23:27.978173 32428 tablet_service.cc:1467] Processing CreateTablet for tablet aa280d05368c4347b3a1c7596e7f9f16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5600cd7c7db24ef8afd53e0485819738]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 12:23:27.978405 445 tablet_service.cc:1467] Processing CreateTablet for tablet aa280d05368c4347b3a1c7596e7f9f16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5600cd7c7db24ef8afd53e0485819738]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 12:23:27.979315 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa280d05368c4347b3a1c7596e7f9f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:27.979351 445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa280d05368c4347b3a1c7596e7f9f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:27.979377 32428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa280d05368c4347b3a1c7596e7f9f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:27.983429 571 tablet_bootstrap.cc:492] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: Bootstrap starting.
I20260806 12:23:27.983693 570 tablet_bootstrap.cc:492] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe: Bootstrap starting.
I20260806 12:23:27.984282 572 tablet_bootstrap.cc:492] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f: Bootstrap starting.
I20260806 12:23:27.984827 571 tablet_bootstrap.cc:654] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:27.984869 570 tablet_bootstrap.cc:654] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:27.985292 572 tablet_bootstrap.cc:654] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:27.986449 571 tablet_bootstrap.cc:492] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: No bootstrap required, opened a new log
I20260806 12:23:27.986514 570 tablet_bootstrap.cc:492] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe: No bootstrap required, opened a new log
I20260806 12:23:27.986557 571 ts_tablet_manager.cc:1397] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260806 12:23:27.986608 570 ts_tablet_manager.cc:1397] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:23:27.987052 570 raft_consensus.cc:348] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:27.987150 570 raft_consensus.cc:374] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:27.987170 570 raft_consensus.cc:729] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9eafc562da4dfea97fe03db90f27fe, State: Initialized, Role: FOLLOWER
I20260806 12:23:27.987267 571 raft_consensus.cc:348] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:27.987365 571 raft_consensus.cc:374] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:27.987385 571 raft_consensus.cc:729] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21523712491c4b759bc7e48ee84123e1, State: Initialized, Role: FOLLOWER
I20260806 12:23:27.987349 570 consensus_queue.cc:260] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:27.987643 576 tablet_replica.cc:439] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } }
I20260806 12:23:27.987625 571 consensus_queue.cc:260] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:27.987787 576 tablet_replica.cc:442] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:27.988011 571 ts_tablet_manager.cc:1428] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 12:23:27.988200 580 tablet_replica.cc:439] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } }
I20260806 12:23:27.988369 580 tablet_replica.cc:442] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:27.988998 572 tablet_bootstrap.cc:492] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f: No bootstrap required, opened a new log
I20260806 12:23:27.989106 572 ts_tablet_manager.cc:1397] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260806 12:23:27.989261 570 ts_tablet_manager.cc:1428] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:23:27.989681 572 raft_consensus.cc:348] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:27.989850 572 raft_consensus.cc:374] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:27.989907 572 raft_consensus.cc:729] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a718cad601d4fda828474f2d5249b1f, State: Initialized, Role: FOLLOWER
I20260806 12:23:27.990080 572 consensus_queue.cc:260] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:27.990490 579 tablet_replica.cc:439] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } }
I20260806 12:23:27.990687 579 tablet_replica.cc:442] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:27.991003 572 ts_tablet_manager.cc:1428] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260806 12:23:28.073376 580 raft_consensus.cc:482] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:28.073591 580 raft_consensus.cc:504] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:28.075232 580 leader_election.cc:288] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339), 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67:41935)
I20260806 12:23:28.080032 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:28.075726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:44023 (local address 127.28.210.65:33339)
0806 12:23:28.075919 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:28.075925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:28.076349 (+ 424us) server_negotiation.cc:407] Connection header received
0806 12:23:28.076595 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:28.076598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:28.076644 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:28.076714 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:28.077027 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.077585 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.078865 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.079330 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.079458 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:28.079525 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:28.079779 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 12:23:28.079868 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 12:23:28.080047 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:28.075576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:33339 (local address 127.28.210.66:44023)
0806 12:23:28.076235 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:28.076260 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:28.076457 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:28.076742 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:28.076745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:28.076765 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:28.076899 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:28.076906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.077707 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.077711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:28.078696 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:28.078708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.079502 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.079506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:28.079615 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:28.079636 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 12:23:28.079752 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 12:23:28.079842 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":186,"threads_started":1}
I20260806 12:23:28.080070 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:28.075869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:46697 (local address 127.28.210.67:41935)
0806 12:23:28.076069 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:28.076112 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:28.076418 (+ 306us) server_negotiation.cc:407] Connection header received
0806 12:23:28.076663 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:28.076668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:28.076721 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:28.076814 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:28.077154 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.077725 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.078952 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.079444 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.079587 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:28.079690 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:28.079835 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 12:23:28.079918 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 12:23:28.080374 598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:28.075576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:41935 (local address 127.28.210.66:46697)
0806 12:23:28.076235 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:28.076254 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:28.076524 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:28.076837 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:28.076841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:28.076862 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:28.077024 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:28.077031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.077879 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.077884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:28.078750 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:28.078762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.079584 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.079589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:28.079701 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:28.079732 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 12:23:28.079819 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 12:23:28.079905 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":212,"thread_start_us":54,"threads_started":1}
I20260806 12:23:28.080807 32453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa280d05368c4347b3a1c7596e7f9f16" candidate_uuid: "21523712491c4b759bc7e48ee84123e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a718cad601d4fda828474f2d5249b1f" is_pre_election: true
I20260806 12:23:28.081013 32453 raft_consensus.cc:2454] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21523712491c4b759bc7e48ee84123e1 in term 0.
I20260806 12:23:28.081192 470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa280d05368c4347b3a1c7596e7f9f16" candidate_uuid: "21523712491c4b759bc7e48ee84123e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9eafc562da4dfea97fe03db90f27fe" is_pre_election: true
I20260806 12:23:28.081493 470 raft_consensus.cc:2454] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21523712491c4b759bc7e48ee84123e1 in term 0.
I20260806 12:23:28.081562 32582 leader_election.cc:302] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a718cad601d4fda828474f2d5249b1f, 21523712491c4b759bc7e48ee84123e1; no voters:
I20260806 12:23:28.081936 580 raft_consensus.cc:2781] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:23:28.082010 580 raft_consensus.cc:482] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:28.082038 580 raft_consensus.cc:3037] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:28.083175 580 raft_consensus.cc:504] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:28.083607 580 leader_election.cc:288] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [CANDIDATE]: Term 1 election: Requested vote from peers 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339), 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67:41935)
I20260806 12:23:28.084241 470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa280d05368c4347b3a1c7596e7f9f16" candidate_uuid: "21523712491c4b759bc7e48ee84123e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9eafc562da4dfea97fe03db90f27fe"
I20260806 12:23:28.084409 470 raft_consensus.cc:3037] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:28.084424 32453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa280d05368c4347b3a1c7596e7f9f16" candidate_uuid: "21523712491c4b759bc7e48ee84123e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a718cad601d4fda828474f2d5249b1f"
I20260806 12:23:28.084589 32453 raft_consensus.cc:3037] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:28.085740 470 raft_consensus.cc:2454] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21523712491c4b759bc7e48ee84123e1 in term 1.
I20260806 12:23:28.085810 32453 raft_consensus.cc:2454] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21523712491c4b759bc7e48ee84123e1 in term 1.
I20260806 12:23:28.086213 32589 leader_election.cc:302] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a718cad601d4fda828474f2d5249b1f, 21523712491c4b759bc7e48ee84123e1, 9f9eafc562da4dfea97fe03db90f27fe; no voters:
I20260806 12:23:28.086428 580 raft_consensus.cc:2781] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:28.086779 580 raft_consensus.cc:686] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [term 1 LEADER]: Becoming Leader. State: Replica: 21523712491c4b759bc7e48ee84123e1, State: Running, Role: LEADER
I20260806 12:23:28.087195 580 consensus_queue.cc:237] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:28.089901 599 tablet_replica.cc:439] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: TxnStatusTablet state changed. Reason: New leader 21523712491c4b759bc7e48ee84123e1. Latest consensus state: current_term: 1 leader_uuid: "21523712491c4b759bc7e48ee84123e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } }
I20260806 12:23:28.090039 599 tablet_replica.cc:442] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: This TxnStatusTablet replica's current role is: LEADER
I20260806 12:23:28.090339 606 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 12:23:28.090549 606 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 12:23:28.091411 32226 catalog_manager.cc:5495] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21523712491c4b759bc7e48ee84123e1 (127.28.210.66). New cstate: current_term: 1 leader_uuid: "21523712491c4b759bc7e48ee84123e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } health_report { overall_health: UNKNOWN } } }
I20260806 12:23:28.121779 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:28.117263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:35587 (local address 127.0.0.1:55090)
0806 12:23:28.117502 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:28.117520 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:28.117639 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:28.118017 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:28.118020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:28.118028 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 12:23:28.118140 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:28.118145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.119264 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.119267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:28.119920 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:28.119931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.120533 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.120536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:28.120607 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:28.120663 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 12:23:28.121510 (+ 847us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 12:23:28.121522 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 12:23:28.121568 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 12:23:28.121653 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 12:23:28.121795 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:28.117496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.28.210.66:35587)
0806 12:23:28.117645 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:28.117650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:28.117668 (+ 18us) server_negotiation.cc:407] Connection header received
0806 12:23:28.117778 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:28.117781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:28.117855 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:28.117962 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:28.118260 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.119120 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.120036 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.120418 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.120520 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:28.120801 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:28.121415 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:28.121527 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:28.121584 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 12:23:28.121645 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 12:23:28.127599 470 raft_consensus.cc:1261] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Refusing update from remote peer 21523712491c4b759bc7e48ee84123e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:28.127725 32453 raft_consensus.cc:1261] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Refusing update from remote peer 21523712491c4b759bc7e48ee84123e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:28.128598 599 consensus_queue.cc:1035] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:28.128855 580 consensus_queue.cc:1035] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:28.134756 576 tablet_replica.cc:439] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe: TxnStatusTablet state changed. Reason: New leader 21523712491c4b759bc7e48ee84123e1. Latest consensus state: current_term: 1 leader_uuid: "21523712491c4b759bc7e48ee84123e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } }
I20260806 12:23:28.134964 576 tablet_replica.cc:442] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:28.139468 580 tablet_replica.cc:439] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21523712491c4b759bc7e48ee84123e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } }
I20260806 12:23:28.139674 580 tablet_replica.cc:442] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: This TxnStatusTablet replica's current role is: LEADER
I20260806 12:23:28.139638 576 tablet_replica.cc:439] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21523712491c4b759bc7e48ee84123e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } }
I20260806 12:23:28.139775 576 tablet_replica.cc:442] T aa280d05368c4347b3a1c7596e7f9f16 P 9f9eafc562da4dfea97fe03db90f27fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:28.139838 613 mvcc.cc:205] Tried to move back new op lower bound from 7315533857282666496 to 7315533857128419328. Current Snapshot: MvccSnapshot[applied={T|T < 7315533857282666496}]
I20260806 12:23:28.140864 579 tablet_replica.cc:439] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f: TxnStatusTablet state changed. Reason: New leader 21523712491c4b759bc7e48ee84123e1. Latest consensus state: current_term: 1 leader_uuid: "21523712491c4b759bc7e48ee84123e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } }
I20260806 12:23:28.141060 579 tablet_replica.cc:442] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:28.142526 612 mvcc.cc:205] Tried to move back new op lower bound from 7315533857282666496 to 7315533857128419328. Current Snapshot: MvccSnapshot[applied={T|T < 7315533857282666496}]
I20260806 12:23:28.146576 580 tablet_replica.cc:439] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21523712491c4b759bc7e48ee84123e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } }
I20260806 12:23:28.146734 580 tablet_replica.cc:442] T aa280d05368c4347b3a1c7596e7f9f16 P 21523712491c4b759bc7e48ee84123e1: This TxnStatusTablet replica's current role is: LEADER
I20260806 12:23:28.149044 618 mvcc.cc:205] Tried to move back new op lower bound from 7315533857282666496 to 7315533857128419328. Current Snapshot: MvccSnapshot[applied={T|T < 7315533857282666496}]
I20260806 12:23:28.149032 579 tablet_replica.cc:439] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21523712491c4b759bc7e48ee84123e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } } }
I20260806 12:23:28.149969 579 tablet_replica.cc:442] T aa280d05368c4347b3a1c7596e7f9f16 P 1a718cad601d4fda828474f2d5249b1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:23:28.156250 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:28.152448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.28.210.67:41935)
0806 12:23:28.152579 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:28.152584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:28.152616 (+ 32us) server_negotiation.cc:407] Connection header received
0806 12:23:28.152986 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:28.152989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:28.153049 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:28.153143 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:28.153407 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.153927 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.154783 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.155001 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.155102 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:28.155365 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:28.155828 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:28.155934 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:28.156024 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 12:23:28.156106 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 12:23:28.168227 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:28.164318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:55973 (local address 127.28.210.66:35587)
0806 12:23:28.164468 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:28.164481 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:28.164724 (+ 243us) server_negotiation.cc:407] Connection header received
0806 12:23:28.164837 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:28.164840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:28.164891 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:28.164955 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:28.165211 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.165741 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.166982 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.167209 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.167301 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:28.167782 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:28.167864 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:28.167922 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:28.168019 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 12:23:28.168072 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 12:23:28.168292 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:28.164213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:35587 (local address 127.28.210.67:55973)
0806 12:23:28.164592 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:28.164609 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:28.164747 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:28.164977 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:28.164980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:28.164988 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 12:23:28.165128 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:28.165135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.165847 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.165851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:28.166834 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:28.166846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.167403 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.167408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:28.167511 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:28.167579 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 12:23:28.167940 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 12:23:28.167947 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 12:23:28.168012 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 12:23:28.168127 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":119,"threads_started":1}
I20260806 12:23:28.177007 32453 raft_consensus.cc:1261] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Refusing update from remote peer 9f9eafc562da4dfea97fe03db90f27fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:28.177112 32710 raft_consensus.cc:1261] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Refusing update from remote peer 9f9eafc562da4dfea97fe03db90f27fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:28.177640 583 consensus_queue.cc:1035] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:28.177892 576 consensus_queue.cc:1035] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:28.205567 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:28.201575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:48073 (local address 127.28.210.67:41935)
0806 12:23:28.201718 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:28.201722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:28.201737 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:23:28.201850 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:28.201854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:28.201912 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:28.201998 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:28.202304 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.203077 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.204396 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.204617 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.204708 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:28.205035 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:28.205150 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:28.205221 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:28.205364 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 12:23:28.205433 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 12:23:28.205567 634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:28.201254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:41935 (local address 127.28.210.66:48073)
0806 12:23:28.201592 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:28.201608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:28.201724 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:28.202018 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:28.202021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:28.202031 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 12:23:28.202201 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:28.202210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.203252 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.203255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:28.204230 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:28.204244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:28.204742 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:28.204746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:28.204837 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:28.204898 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 12:23:28.205239 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 12:23:28.205246 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 12:23:28.205308 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 12:23:28.205402 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":123,"threads_started":1}
I20260806 12:23:28.245568 32226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:28.245841 32226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:28.249894 32226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0: Sending DeleteTablet for 3 replicas of tablet 446fc33a158d436cbbc1aa331996fef8
I20260806 12:23:28.251515 445 tablet_service.cc:1514] Processing DeleteTablet for tablet 446fc33a158d436cbbc1aa331996fef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:28 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:44526
I20260806 12:23:28.251842 32428 tablet_service.cc:1514] Processing DeleteTablet for tablet 446fc33a158d436cbbc1aa331996fef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:28 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:38030
I20260806 12:23:28.252240 638 tablet_replica.cc:330] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe: stopping tablet replica
I20260806 12:23:28.252710 638 raft_consensus.cc:2229] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:23:28.252988 32667 tablet_service.cc:1514] Processing DeleteTablet for tablet 446fc33a158d436cbbc1aa331996fef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:28 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:55078
I20260806 12:23:28.253301 638 raft_consensus.cc:2258] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:28.255254 638 ts_tablet_manager.cc:1905] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:28.255403 32227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:23:28.258349 32227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:28.260339 638 ts_tablet_manager.cc:1918] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:23:28.260488 638 log.cc:1196] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/446fc33a158d436cbbc1aa331996fef8
I20260806 12:23:28.260828 638 ts_tablet_manager.cc:1939] T 446fc33a158d436cbbc1aa331996fef8 P 9f9eafc562da4dfea97fe03db90f27fe: Deleting consensus metadata
I20260806 12:23:28.262383 32197 catalog_manager.cc:4841] TS 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67:41935): tablet 446fc33a158d436cbbc1aa331996fef8 (table test-table [id=5c681069e71d41109bdcd055274def7b]) successfully deleted
I20260806 12:23:28.265019 641 tablet_replica.cc:330] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1: stopping tablet replica
I20260806 12:23:28.265461 641 raft_consensus.cc:2229] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:28.265657 641 raft_consensus.cc:2258] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:28.265592 445 tablet_service.cc:1467] Processing CreateTablet for tablet b8b3e5722de645cab25db08f549db363 (DEFAULT_TABLE table=test-table [id=6733f23870344facbd8c40ed6bd30a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:28.265933 445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b3e5722de645cab25db08f549db363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:28.266036 640 tablet_replica.cc:330] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f: stopping tablet replica
I20260806 12:23:28.266366 640 raft_consensus.cc:2229] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:28.266594 640 raft_consensus.cc:2258] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:28.268812 641 ts_tablet_manager.cc:1905] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:28.269488 32428 tablet_service.cc:1467] Processing CreateTablet for tablet b8b3e5722de645cab25db08f549db363 (DEFAULT_TABLE table=test-table [id=6733f23870344facbd8c40ed6bd30a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:28.270008 32428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b3e5722de645cab25db08f549db363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:28.272286 641 ts_tablet_manager.cc:1918] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:23:28.272408 641 log.cc:1196] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/446fc33a158d436cbbc1aa331996fef8
I20260806 12:23:28.272725 641 ts_tablet_manager.cc:1939] T 446fc33a158d436cbbc1aa331996fef8 P 21523712491c4b759bc7e48ee84123e1: Deleting consensus metadata
I20260806 12:23:28.274427 32193 catalog_manager.cc:4841] TS 21523712491c4b759bc7e48ee84123e1 (127.28.210.66:35587): tablet 446fc33a158d436cbbc1aa331996fef8 (table test-table [id=5c681069e71d41109bdcd055274def7b]) successfully deleted
I20260806 12:23:28.274518 32667 tablet_service.cc:1467] Processing CreateTablet for tablet b8b3e5722de645cab25db08f549db363 (DEFAULT_TABLE table=test-table [id=6733f23870344facbd8c40ed6bd30a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:28.274832 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b3e5722de645cab25db08f549db363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:28.278319 571 tablet_bootstrap.cc:492] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1: Bootstrap starting.
I20260806 12:23:28.279525 570 tablet_bootstrap.cc:492] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe: Bootstrap starting.
I20260806 12:23:28.279578 571 tablet_bootstrap.cc:654] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:28.280809 570 tablet_bootstrap.cc:654] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:28.280834 572 tablet_bootstrap.cc:492] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f: Bootstrap starting.
I20260806 12:23:28.282083 572 tablet_bootstrap.cc:654] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:28.282473 640 ts_tablet_manager.cc:1905] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:28.284983 571 tablet_bootstrap.cc:492] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1: No bootstrap required, opened a new log
I20260806 12:23:28.285143 571 ts_tablet_manager.cc:1397] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20260806 12:23:28.285838 571 raft_consensus.cc:348] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:28.286012 571 raft_consensus.cc:374] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:28.286054 571 raft_consensus.cc:729] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21523712491c4b759bc7e48ee84123e1, State: Initialized, Role: FOLLOWER
I20260806 12:23:28.286243 571 consensus_queue.cc:260] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:28.286587 571 ts_tablet_manager.cc:1428] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260806 12:23:28.286585 640 ts_tablet_manager.cc:1918] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:23:28.286716 640 log.cc:1196] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/wal/wals/446fc33a158d436cbbc1aa331996fef8
I20260806 12:23:28.287232 640 ts_tablet_manager.cc:1939] T 446fc33a158d436cbbc1aa331996fef8 P 1a718cad601d4fda828474f2d5249b1f: Deleting consensus metadata
I20260806 12:23:28.289317 32193 catalog_manager.cc:4841] TS 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339): tablet 446fc33a158d436cbbc1aa331996fef8 (table test-table [id=5c681069e71d41109bdcd055274def7b]) successfully deleted
I20260806 12:23:28.291049 570 tablet_bootstrap.cc:492] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe: No bootstrap required, opened a new log
I20260806 12:23:28.291049 572 tablet_bootstrap.cc:492] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f: No bootstrap required, opened a new log
I20260806 12:23:28.291183 570 ts_tablet_manager.cc:1397] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20260806 12:23:28.291198 572 ts_tablet_manager.cc:1397] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260806 12:23:28.291754 572 raft_consensus.cc:348] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:28.291754 570 raft_consensus.cc:348] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:28.291908 572 raft_consensus.cc:374] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:28.291908 570 raft_consensus.cc:374] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:28.291949 570 raft_consensus.cc:729] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9eafc562da4dfea97fe03db90f27fe, State: Initialized, Role: FOLLOWER
I20260806 12:23:28.291949 572 raft_consensus.cc:729] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a718cad601d4fda828474f2d5249b1f, State: Initialized, Role: FOLLOWER
I20260806 12:23:28.292150 572 consensus_queue.cc:260] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:28.292174 570 consensus_queue.cc:260] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:28.292512 572 ts_tablet_manager.cc:1428] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 12:23:28.293962 570 ts_tablet_manager.cc:1428] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 12:23:28.372848 583 raft_consensus.cc:482] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:28.373031 583 raft_consensus.cc:504] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:28.373569 583 leader_election.cc:288] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339), 21523712491c4b759bc7e48ee84123e1 (127.28.210.66:35587)
I20260806 12:23:28.374183 32453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b3e5722de645cab25db08f549db363" candidate_uuid: "9f9eafc562da4dfea97fe03db90f27fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a718cad601d4fda828474f2d5249b1f" is_pre_election: true
I20260806 12:23:28.374184 32710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b3e5722de645cab25db08f549db363" candidate_uuid: "9f9eafc562da4dfea97fe03db90f27fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21523712491c4b759bc7e48ee84123e1" is_pre_election: true
I20260806 12:23:28.374365 32453 raft_consensus.cc:2454] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9eafc562da4dfea97fe03db90f27fe in term 0.
I20260806 12:23:28.374365 32710 raft_consensus.cc:2454] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9eafc562da4dfea97fe03db90f27fe in term 0.
I20260806 12:23:28.374784 360 leader_election.cc:302] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21523712491c4b759bc7e48ee84123e1, 9f9eafc562da4dfea97fe03db90f27fe; no voters:
I20260806 12:23:28.375308 583 raft_consensus.cc:2781] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:23:28.375388 583 raft_consensus.cc:482] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:28.375434 583 raft_consensus.cc:3037] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:28.376690 583 raft_consensus.cc:504] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:28.377138 583 leader_election.cc:288] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [CANDIDATE]: Term 1 election: Requested vote from peers 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339), 21523712491c4b759bc7e48ee84123e1 (127.28.210.66:35587)
I20260806 12:23:28.377699 32453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b3e5722de645cab25db08f549db363" candidate_uuid: "9f9eafc562da4dfea97fe03db90f27fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a718cad601d4fda828474f2d5249b1f"
I20260806 12:23:28.377699 32710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b3e5722de645cab25db08f549db363" candidate_uuid: "9f9eafc562da4dfea97fe03db90f27fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21523712491c4b759bc7e48ee84123e1"
I20260806 12:23:28.377853 32453 raft_consensus.cc:3037] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:28.377854 32710 raft_consensus.cc:3037] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:28.379117 32453 raft_consensus.cc:2454] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9eafc562da4dfea97fe03db90f27fe in term 1.
I20260806 12:23:28.379117 32710 raft_consensus.cc:2454] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9eafc562da4dfea97fe03db90f27fe in term 1.
I20260806 12:23:28.379653 360 leader_election.cc:302] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a718cad601d4fda828474f2d5249b1f, 9f9eafc562da4dfea97fe03db90f27fe; no voters:
I20260806 12:23:28.379952 583 raft_consensus.cc:2781] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:28.380206 583 raft_consensus.cc:686] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 1 LEADER]: Becoming Leader. State: Replica: 9f9eafc562da4dfea97fe03db90f27fe, State: Running, Role: LEADER
I20260806 12:23:28.380394 583 consensus_queue.cc:237] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:28.382491 32227 catalog_manager.cc:5495] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67). New cstate: current_term: 1 leader_uuid: "9f9eafc562da4dfea97fe03db90f27fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } health_report { overall_health: HEALTHY } } }
I20260806 12:23:28.408444 32710 raft_consensus.cc:1261] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Refusing update from remote peer 9f9eafc562da4dfea97fe03db90f27fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:28.408444 32453 raft_consensus.cc:1261] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Refusing update from remote peer 9f9eafc562da4dfea97fe03db90f27fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:28.409035 583 consensus_queue.cc:1035] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:28.409266 631 consensus_queue.cc:1035] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 12:23:28.430411 613 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 12:23:30.020020 32227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:30.020255 32227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:30.023322 32227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0: Sending DeleteTablet for 3 replicas of tablet b8b3e5722de645cab25db08f549db363
I20260806 12:23:30.024201 32428 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b3e5722de645cab25db08f549db363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:30 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:38030
I20260806 12:23:30.024192 445 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b3e5722de645cab25db08f549db363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:30 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:44526
I20260806 12:23:30.024212 32667 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b3e5722de645cab25db08f549db363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:30 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:55078
I20260806 12:23:30.024626 681 tablet_replica.cc:330] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1: stopping tablet replica
I20260806 12:23:30.024771 684 tablet_replica.cc:330] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe: stopping tablet replica
I20260806 12:23:30.024843 681 raft_consensus.cc:2229] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:30.024971 684 raft_consensus.cc:2229] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:23:30.025043 681 raft_consensus.cc:2258] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:30.025245 684 raft_consensus.cc:2258] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:30.025897 684 ts_tablet_manager.cc:1905] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:30.025898 681 ts_tablet_manager.cc:1905] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:30.028108 683 tablet_replica.cc:330] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f: stopping tablet replica
I20260806 12:23:30.028890 683 raft_consensus.cc:2229] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:30.029124 683 raft_consensus.cc:2258] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:30.030042 683 ts_tablet_manager.cc:1905] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:30.030460 684 ts_tablet_manager.cc:1918] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 12:23:30.030589 684 log.cc:1196] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/b8b3e5722de645cab25db08f549db363
I20260806 12:23:30.031117 681 ts_tablet_manager.cc:1918] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 12:23:30.031273 681 log.cc:1196] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/b8b3e5722de645cab25db08f549db363
I20260806 12:23:30.031467 684 ts_tablet_manager.cc:1939] T b8b3e5722de645cab25db08f549db363 P 9f9eafc562da4dfea97fe03db90f27fe: Deleting consensus metadata
I20260806 12:23:30.031574 681 ts_tablet_manager.cc:1939] T b8b3e5722de645cab25db08f549db363 P 21523712491c4b759bc7e48ee84123e1: Deleting consensus metadata
I20260806 12:23:30.032462 32197 catalog_manager.cc:4841] TS 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67:41935): tablet b8b3e5722de645cab25db08f549db363 (table test-table [id=6733f23870344facbd8c40ed6bd30a8d]) successfully deleted
I20260806 12:23:30.032462 32193 catalog_manager.cc:4841] TS 21523712491c4b759bc7e48ee84123e1 (127.28.210.66:35587): tablet b8b3e5722de645cab25db08f549db363 (table test-table [id=6733f23870344facbd8c40ed6bd30a8d]) successfully deleted
I20260806 12:23:30.032900 683 ts_tablet_manager.cc:1918] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 12:23:30.033003 683 log.cc:1196] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/wal/wals/b8b3e5722de645cab25db08f549db363
I20260806 12:23:30.033385 683 ts_tablet_manager.cc:1939] T b8b3e5722de645cab25db08f549db363 P 1a718cad601d4fda828474f2d5249b1f: Deleting consensus metadata
I20260806 12:23:30.034467 32193 catalog_manager.cc:4841] TS 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339): tablet b8b3e5722de645cab25db08f549db363 (table test-table [id=6733f23870344facbd8c40ed6bd30a8d]) successfully deleted
Aug 06 12:23:30 dist-test-slave-lgr5 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019010, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 12:23:30 dist-test-slave-lgr5 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019010, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:30.062500 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.050278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.28.210.126:39895)
0806 12:23:30.050595 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:30.050602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:30.050617 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:23:30.050670 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:30.050673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:30.050719 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:30.050842 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:30.051813 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.052552 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.053679 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.054152 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.054263 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.057579 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:30.057585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:30.057587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:30.057633 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:30.059589 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:30.060657 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:30.060662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:30.060664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:30.060817 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:30.061414 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:30.061418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:30.061420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:30.061690 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:30.061847 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:30.062176 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 12:23:30.062328 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":126,"threads_started":1}
I20260806 12:23:30.066851 32227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:23:30.067423 32227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:30.075255 32428 tablet_service.cc:1467] Processing CreateTablet for tablet 1b46865372e84a70aad1ffb9ca09750e (DEFAULT_TABLE table=test-table [id=529f430c11864383af9ca8c87be9a16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:30.075333 445 tablet_service.cc:1467] Processing CreateTablet for tablet 1b46865372e84a70aad1ffb9ca09750e (DEFAULT_TABLE table=test-table [id=529f430c11864383af9ca8c87be9a16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:30.075659 32428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b46865372e84a70aad1ffb9ca09750e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:30.075673 445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b46865372e84a70aad1ffb9ca09750e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:30.075757 32667 tablet_service.cc:1467] Processing CreateTablet for tablet 1b46865372e84a70aad1ffb9ca09750e (DEFAULT_TABLE table=test-table [id=529f430c11864383af9ca8c87be9a16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:30.076161 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b46865372e84a70aad1ffb9ca09750e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:30.078794 701 tablet_bootstrap.cc:492] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f: Bootstrap starting.
I20260806 12:23:30.079481 704 tablet_bootstrap.cc:492] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1: Bootstrap starting.
I20260806 12:23:30.079491 703 tablet_bootstrap.cc:492] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe: Bootstrap starting.
I20260806 12:23:30.080156 701 tablet_bootstrap.cc:654] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:30.080646 704 tablet_bootstrap.cc:654] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:30.080771 703 tablet_bootstrap.cc:654] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:30.082034 701 tablet_bootstrap.cc:492] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f: No bootstrap required, opened a new log
I20260806 12:23:30.082151 701 ts_tablet_manager.cc:1397] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 12:23:30.082295 703 tablet_bootstrap.cc:492] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe: No bootstrap required, opened a new log
I20260806 12:23:30.082295 704 tablet_bootstrap.cc:492] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1: No bootstrap required, opened a new log
I20260806 12:23:30.082414 703 ts_tablet_manager.cc:1397] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:23:30.082414 704 ts_tablet_manager.cc:1397] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:23:30.082906 701 raft_consensus.cc:348] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.082914 704 raft_consensus.cc:348] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.082914 703 raft_consensus.cc:348] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.083037 701 raft_consensus.cc:374] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:30.083047 704 raft_consensus.cc:374] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:30.083047 703 raft_consensus.cc:374] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:30.083073 701 raft_consensus.cc:729] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a718cad601d4fda828474f2d5249b1f, State: Initialized, Role: FOLLOWER
I20260806 12:23:30.083084 703 raft_consensus.cc:729] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9eafc562da4dfea97fe03db90f27fe, State: Initialized, Role: FOLLOWER
I20260806 12:23:30.083084 704 raft_consensus.cc:729] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21523712491c4b759bc7e48ee84123e1, State: Initialized, Role: FOLLOWER
I20260806 12:23:30.083233 701 consensus_queue.cc:260] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.083240 703 consensus_queue.cc:260] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.083268 704 consensus_queue.cc:260] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.083559 704 ts_tablet_manager.cc:1428] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 12:23:30.083559 703 ts_tablet_manager.cc:1428] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 12:23:30.083655 701 ts_tablet_manager.cc:1428] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260806 12:23:30.214890 711 raft_consensus.cc:482] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:30.215087 711 raft_consensus.cc:504] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.216347 711 leader_election.cc:288] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21523712491c4b759bc7e48ee84123e1 (127.28.210.66:35587), 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67:41935)
I20260806 12:23:30.222344 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.216681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:41935 (local address 127.28.210.65:59045)
0806 12:23:30.217144 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:30.217160 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:30.217323 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:30.217673 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:30.217676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:30.217712 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:30.217956 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:30.217962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.219087 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.219091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:30.220749 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:30.220766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.221732 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.221736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:30.221900 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.221950 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 12:23:30.222035 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 12:23:30.222132 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":195,"spinlock_wait_cycles":27264,"thread_start_us":160,"threads_started":1}
I20260806 12:23:30.222545 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.216897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:45769 (local address 127.28.210.66:35587)
0806 12:23:30.217204 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:30.217215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:30.217256 (+ 41us) server_negotiation.cc:407] Connection header received
0806 12:23:30.217444 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:30.217447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:30.217503 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:30.217613 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:30.218085 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.218929 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.220913 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.221585 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.221730 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.221795 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:30.222182 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 12:23:30.222309 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":116,"threads_started":1}
I20260806 12:23:30.222391 713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.216682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:35587 (local address 127.28.210.65:45769)
0806 12:23:30.217177 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:30.217197 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:30.217322 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:30.217673 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:30.217676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:30.217712 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:30.217950 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:30.217959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.219087 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.219091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:30.220749 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:30.220766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.221732 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.221736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:30.221886 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.221951 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 12:23:30.222131 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 12:23:30.222188 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":238,"spinlock_wait_cycles":22912,"thread_start_us":172,"threads_started":1}
I20260806 12:23:30.223376 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.216819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:59045 (local address 127.28.210.67:41935)
0806 12:23:30.217204 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:30.217215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:30.217241 (+ 26us) server_negotiation.cc:407] Connection header received
0806 12:23:30.217444 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:30.217447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:30.217503 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:30.217613 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:30.218086 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.218929 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.220913 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.221587 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.221731 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.221795 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:30.222065 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 12:23:30.223240 (+ 1175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":89,"threads_started":1}
I20260806 12:23:30.223474 32710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b46865372e84a70aad1ffb9ca09750e" candidate_uuid: "1a718cad601d4fda828474f2d5249b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21523712491c4b759bc7e48ee84123e1" is_pre_election: true
I20260806 12:23:30.223665 32710 raft_consensus.cc:2454] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a718cad601d4fda828474f2d5249b1f in term 0.
I20260806 12:23:30.223805 470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b46865372e84a70aad1ffb9ca09750e" candidate_uuid: "1a718cad601d4fda828474f2d5249b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9eafc562da4dfea97fe03db90f27fe" is_pre_election: true
I20260806 12:23:30.223976 470 raft_consensus.cc:2454] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a718cad601d4fda828474f2d5249b1f in term 0.
I20260806 12:23:30.224164 32351 leader_election.cc:302] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a718cad601d4fda828474f2d5249b1f, 21523712491c4b759bc7e48ee84123e1; no voters:
I20260806 12:23:30.224494 711 raft_consensus.cc:2781] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:23:30.224566 711 raft_consensus.cc:482] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:30.224619 711 raft_consensus.cc:3037] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:30.225724 711 raft_consensus.cc:504] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.226121 711 leader_election.cc:288] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [CANDIDATE]: Term 1 election: Requested vote from peers 21523712491c4b759bc7e48ee84123e1 (127.28.210.66:35587), 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67:41935)
I20260806 12:23:30.226605 470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b46865372e84a70aad1ffb9ca09750e" candidate_uuid: "1a718cad601d4fda828474f2d5249b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9eafc562da4dfea97fe03db90f27fe"
I20260806 12:23:30.226584 32710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b46865372e84a70aad1ffb9ca09750e" candidate_uuid: "1a718cad601d4fda828474f2d5249b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21523712491c4b759bc7e48ee84123e1"
I20260806 12:23:30.226735 32710 raft_consensus.cc:3037] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:30.226742 470 raft_consensus.cc:3037] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:30.227804 32710 raft_consensus.cc:2454] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a718cad601d4fda828474f2d5249b1f in term 1.
I20260806 12:23:30.227818 470 raft_consensus.cc:2454] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a718cad601d4fda828474f2d5249b1f in term 1.
I20260806 12:23:30.228221 32351 leader_election.cc:302] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a718cad601d4fda828474f2d5249b1f, 21523712491c4b759bc7e48ee84123e1; no voters:
I20260806 12:23:30.228467 711 raft_consensus.cc:2781] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:30.228717 711 raft_consensus.cc:686] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 1 LEADER]: Becoming Leader. State: Replica: 1a718cad601d4fda828474f2d5249b1f, State: Running, Role: LEADER
I20260806 12:23:30.229050 711 consensus_queue.cc:237] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.232362 32226 catalog_manager.cc:5495] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a718cad601d4fda828474f2d5249b1f (127.28.210.65). New cstate: current_term: 1 leader_uuid: "1a718cad601d4fda828474f2d5249b1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } health_report { overall_health: UNKNOWN } } }
I20260806 12:23:30.287468 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.283066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.28.210.65:33339)
0806 12:23:30.283403 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:30.283411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:30.283423 (+ 12us) server_negotiation.cc:407] Connection header received
0806 12:23:30.283535 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:30.283537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:30.283598 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:30.283725 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:30.284025 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.284746 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.285720 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.286026 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.286131 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.286468 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:30.286995 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:30.287106 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:30.287186 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 12:23:30.287252 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":126,"threads_started":1}
I20260806 12:23:30.299965 732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.295932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:35587 (local address 127.28.210.65:48895)
0806 12:23:30.296516 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:30.296533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:30.296658 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:30.296942 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:30.296945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:30.296953 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 12:23:30.297099 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:30.297104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.297867 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.297871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:30.298561 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:30.298570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.299057 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.299060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:30.299126 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.299186 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 12:23:30.299595 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 12:23:30.299604 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 12:23:30.299680 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 12:23:30.299765 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":139,"threads_started":1}
I20260806 12:23:30.300005 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.296035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:48895 (local address 127.28.210.66:35587)
0806 12:23:30.296188 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:30.296192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:30.296642 (+ 450us) server_negotiation.cc:407] Connection header received
0806 12:23:30.296755 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:30.296757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:30.296819 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:30.296896 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:30.297194 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.297738 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.298701 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.298934 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.299029 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.299363 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:30.299449 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:30.299516 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:30.299752 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 12:23:30.299816 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 12:23:30.308514 32710 raft_consensus.cc:1261] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Refusing update from remote peer 1a718cad601d4fda828474f2d5249b1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:30.308544 470 raft_consensus.cc:1261] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Refusing update from remote peer 1a718cad601d4fda828474f2d5249b1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:30.309104 722 consensus_queue.cc:1035] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:30.309331 711 consensus_queue.cc:1035] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:30.331811 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.327968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:42649 (local address 127.28.210.65:33339)
0806 12:23:30.328121 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:30.328126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:30.328513 (+ 387us) server_negotiation.cc:407] Connection header received
0806 12:23:30.328631 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:30.328633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:30.328678 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:30.328739 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:30.328949 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.329541 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.330616 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.330928 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.331044 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.331304 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:30.331417 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:30.331486 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:30.331595 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 12:23:30.331666 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 12:23:30.331816 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.327894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:33339 (local address 127.28.210.66:42649)
0806 12:23:30.328412 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:30.328430 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:30.328540 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:30.328755 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:30.328757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:30.328763 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 12:23:30.328874 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:30.328878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.329669 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.329673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:30.330475 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:30.330491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.331053 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.331056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:30.331144 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.331184 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 12:23:30.331506 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 12:23:30.331513 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 12:23:30.331573 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 12:23:30.331674 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":196,"threads_started":1}
I20260806 12:23:30.361428 32226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:30.361635 32226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:30.364635 32226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0: Sending DeleteTablet for 3 replicas of tablet 1b46865372e84a70aad1ffb9ca09750e
I20260806 12:23:30.365569 32667 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b46865372e84a70aad1ffb9ca09750e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:30 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:55078
I20260806 12:23:30.365571 32428 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b46865372e84a70aad1ffb9ca09750e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:30 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:38030
I20260806 12:23:30.365643 445 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b46865372e84a70aad1ffb9ca09750e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:30 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:44526
I20260806 12:23:30.365825 681 tablet_replica.cc:330] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1: stopping tablet replica
I20260806 12:23:30.365835 683 tablet_replica.cc:330] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f: stopping tablet replica
I20260806 12:23:30.365967 681 raft_consensus.cc:2229] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:30.365976 683 raft_consensus.cc:2229] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:23:30.366128 681 raft_consensus.cc:2258] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:30.366406 683 raft_consensus.cc:2258] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:30.366662 684 tablet_replica.cc:330] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe: stopping tablet replica
I20260806 12:23:30.366822 684 raft_consensus.cc:2229] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:30.366971 684 raft_consensus.cc:2258] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:30.367715 681 ts_tablet_manager.cc:1905] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:30.367688 32226 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:23:30.368296 32226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:30.368824 683 ts_tablet_manager.cc:1905] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:30.368054 684 ts_tablet_manager.cc:1905] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:30.373121 683 ts_tablet_manager.cc:1918] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:23:30.373260 683 log.cc:1196] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-0/wal/wals/1b46865372e84a70aad1ffb9ca09750e
I20260806 12:23:30.373575 683 ts_tablet_manager.cc:1939] T 1b46865372e84a70aad1ffb9ca09750e P 1a718cad601d4fda828474f2d5249b1f: Deleting consensus metadata
I20260806 12:23:30.374181 684 ts_tablet_manager.cc:1918] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:23:30.374265 684 log.cc:1196] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/1b46865372e84a70aad1ffb9ca09750e
I20260806 12:23:30.374529 684 ts_tablet_manager.cc:1939] T 1b46865372e84a70aad1ffb9ca09750e P 9f9eafc562da4dfea97fe03db90f27fe: Deleting consensus metadata
I20260806 12:23:30.374751 32193 catalog_manager.cc:4841] TS 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339): tablet 1b46865372e84a70aad1ffb9ca09750e (table test-table [id=529f430c11864383af9ca8c87be9a16d]) successfully deleted
I20260806 12:23:30.375656 32197 catalog_manager.cc:4841] TS 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67:41935): tablet 1b46865372e84a70aad1ffb9ca09750e (table test-table [id=529f430c11864383af9ca8c87be9a16d]) successfully deleted
I20260806 12:23:30.377256 681 ts_tablet_manager.cc:1918] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:23:30.377471 681 log.cc:1196] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/1b46865372e84a70aad1ffb9ca09750e
I20260806 12:23:30.377810 681 ts_tablet_manager.cc:1939] T 1b46865372e84a70aad1ffb9ca09750e P 21523712491c4b759bc7e48ee84123e1: Deleting consensus metadata
I20260806 12:23:30.379315 32193 catalog_manager.cc:4841] TS 21523712491c4b759bc7e48ee84123e1 (127.28.210.66:35587): tablet 1b46865372e84a70aad1ffb9ca09750e (table test-table [id=529f430c11864383af9ca8c87be9a16d]) successfully deleted
I20260806 12:23:30.380227 32428 tablet_service.cc:1467] Processing CreateTablet for tablet fd3989e971f34164a9e4571acd600eef (DEFAULT_TABLE table=test-table [id=90562c067d2949a7bc7b4613883a4c6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:30.380483 445 tablet_service.cc:1467] Processing CreateTablet for tablet fd3989e971f34164a9e4571acd600eef (DEFAULT_TABLE table=test-table [id=90562c067d2949a7bc7b4613883a4c6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:30.380611 32428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd3989e971f34164a9e4571acd600eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:30.380829 445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd3989e971f34164a9e4571acd600eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:30.384145 701 tablet_bootstrap.cc:492] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f: Bootstrap starting.
I20260806 12:23:30.385107 32667 tablet_service.cc:1467] Processing CreateTablet for tablet fd3989e971f34164a9e4571acd600eef (DEFAULT_TABLE table=test-table [id=90562c067d2949a7bc7b4613883a4c6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:30.385527 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd3989e971f34164a9e4571acd600eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:30.385526 703 tablet_bootstrap.cc:492] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe: Bootstrap starting.
I20260806 12:23:30.386691 701 tablet_bootstrap.cc:654] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:30.386703 703 tablet_bootstrap.cc:654] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:30.388810 701 tablet_bootstrap.cc:492] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f: No bootstrap required, opened a new log
I20260806 12:23:30.388808 704 tablet_bootstrap.cc:492] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1: Bootstrap starting.
I20260806 12:23:30.388937 701 ts_tablet_manager.cc:1397] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 12:23:30.389977 704 tablet_bootstrap.cc:654] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:30.389958 701 raft_consensus.cc:348] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.390112 701 raft_consensus.cc:374] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:30.390153 701 raft_consensus.cc:729] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a718cad601d4fda828474f2d5249b1f, State: Initialized, Role: FOLLOWER
I20260806 12:23:30.390300 701 consensus_queue.cc:260] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.390669 701 ts_tablet_manager.cc:1428] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260806 12:23:30.393162 703 tablet_bootstrap.cc:492] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe: No bootstrap required, opened a new log
I20260806 12:23:30.393301 703 ts_tablet_manager.cc:1397] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260806 12:23:30.393836 703 raft_consensus.cc:348] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.393992 703 raft_consensus.cc:374] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:30.394034 703 raft_consensus.cc:729] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9eafc562da4dfea97fe03db90f27fe, State: Initialized, Role: FOLLOWER
I20260806 12:23:30.394172 703 consensus_queue.cc:260] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.394477 703 ts_tablet_manager.cc:1428] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 12:23:30.395845 704 tablet_bootstrap.cc:492] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1: No bootstrap required, opened a new log
I20260806 12:23:30.395975 704 ts_tablet_manager.cc:1397] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260806 12:23:30.396533 704 raft_consensus.cc:348] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.396682 704 raft_consensus.cc:374] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:30.396724 704 raft_consensus.cc:729] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21523712491c4b759bc7e48ee84123e1, State: Initialized, Role: FOLLOWER
I20260806 12:23:30.396862 704 consensus_queue.cc:260] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.397182 704 ts_tablet_manager.cc:1428] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 12:23:30.480058 724 raft_consensus.cc:482] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:30.480289 724 raft_consensus.cc:504] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.480870 724 leader_election.cc:288] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339), 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67:41935)
I20260806 12:23:30.481299 32453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd3989e971f34164a9e4571acd600eef" candidate_uuid: "21523712491c4b759bc7e48ee84123e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a718cad601d4fda828474f2d5249b1f" is_pre_election: true
I20260806 12:23:30.481515 32453 raft_consensus.cc:2454] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21523712491c4b759bc7e48ee84123e1 in term 0.
I20260806 12:23:30.481465 470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd3989e971f34164a9e4571acd600eef" candidate_uuid: "21523712491c4b759bc7e48ee84123e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9eafc562da4dfea97fe03db90f27fe" is_pre_election: true
I20260806 12:23:30.481645 470 raft_consensus.cc:2454] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21523712491c4b759bc7e48ee84123e1 in term 0.
I20260806 12:23:30.481878 32582 leader_election.cc:302] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a718cad601d4fda828474f2d5249b1f, 21523712491c4b759bc7e48ee84123e1; no voters:
I20260806 12:23:30.482115 724 raft_consensus.cc:2781] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:23:30.482203 724 raft_consensus.cc:482] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:30.482242 724 raft_consensus.cc:3037] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:30.483743 724 raft_consensus.cc:504] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.484292 724 leader_election.cc:288] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [CANDIDATE]: Term 1 election: Requested vote from peers 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339), 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67:41935)
I20260806 12:23:30.484810 32453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd3989e971f34164a9e4571acd600eef" candidate_uuid: "21523712491c4b759bc7e48ee84123e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a718cad601d4fda828474f2d5249b1f"
I20260806 12:23:30.484813 470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd3989e971f34164a9e4571acd600eef" candidate_uuid: "21523712491c4b759bc7e48ee84123e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9eafc562da4dfea97fe03db90f27fe"
I20260806 12:23:30.484977 32453 raft_consensus.cc:3037] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:30.484997 470 raft_consensus.cc:3037] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:30.486356 32453 raft_consensus.cc:2454] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21523712491c4b759bc7e48ee84123e1 in term 1.
I20260806 12:23:30.486486 470 raft_consensus.cc:2454] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21523712491c4b759bc7e48ee84123e1 in term 1.
I20260806 12:23:30.486812 32582 leader_election.cc:302] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a718cad601d4fda828474f2d5249b1f, 21523712491c4b759bc7e48ee84123e1; no voters:
I20260806 12:23:30.487061 724 raft_consensus.cc:2781] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:30.487495 724 raft_consensus.cc:686] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 1 LEADER]: Becoming Leader. State: Replica: 21523712491c4b759bc7e48ee84123e1, State: Running, Role: LEADER
I20260806 12:23:30.487676 724 consensus_queue.cc:237] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } }
I20260806 12:23:30.490532 32217 catalog_manager.cc:5495] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21523712491c4b759bc7e48ee84123e1 (127.28.210.66). New cstate: current_term: 1 leader_uuid: "21523712491c4b759bc7e48ee84123e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21523712491c4b759bc7e48ee84123e1" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 35587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 } health_report { overall_health: UNKNOWN } } }
I20260806 12:23:30.517802 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.514477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.28.210.66:35587)
0806 12:23:30.514612 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:30.514617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:30.514632 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:23:30.514792 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:30.514795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:30.514848 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:30.514914 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:30.515193 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.515740 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.516669 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.516900 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.517007 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.517277 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:30.517367 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:30.517432 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:30.517560 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 12:23:30.517623 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 12:23:30.532420 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.525101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:42055 (local address 127.28.210.66:35587)
0806 12:23:30.525251 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:30.525256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:30.528292 (+ 3036us) server_negotiation.cc:407] Connection header received
0806 12:23:30.528465 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:30.528468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:30.528531 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:30.528635 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:30.528979 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.529836 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.531027 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.531397 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.531521 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.531810 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:30.531959 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:30.532046 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:30.532161 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 12:23:30.532229 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":51}
I20260806 12:23:30.532419 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:30.524890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:35587 (local address 127.28.210.66:42055)
0806 12:23:30.528168 (+ 3278us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:30.528189 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:30.528348 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:30.528676 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:30.528680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:30.528690 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 12:23:30.528847 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:30.528855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.529984 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.529988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:30.530875 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:30.530888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:30.531547 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:30.531550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:30.531633 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:30.531676 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 12:23:30.532055 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 12:23:30.532061 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 12:23:30.532146 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 12:23:30.532251 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260806 12:23:30.541709 469 raft_consensus.cc:1261] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Refusing update from remote peer 21523712491c4b759bc7e48ee84123e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:30.542515 32453 raft_consensus.cc:1261] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Refusing update from remote peer 21523712491c4b759bc7e48ee84123e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:30.542548 724 consensus_queue.cc:1035] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9eafc562da4dfea97fe03db90f27fe" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:30.543586 657 consensus_queue.cc:1035] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a718cad601d4fda828474f2d5249b1f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 12:23:30.564082 655 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 12:23:32.142552 32217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:32.142746 32217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:32.145711 32217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36d7c70870da48f2a1956a6c961e5be0: Sending DeleteTablet for 3 replicas of tablet fd3989e971f34164a9e4571acd600eef
I20260806 12:23:32.146463 32428 tablet_service.cc:1514] Processing DeleteTablet for tablet fd3989e971f34164a9e4571acd600eef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:32 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:38030
I20260806 12:23:32.146500 32667 tablet_service.cc:1514] Processing DeleteTablet for tablet fd3989e971f34164a9e4571acd600eef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:32 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:55078
I20260806 12:23:32.146517 445 tablet_service.cc:1514] Processing DeleteTablet for tablet fd3989e971f34164a9e4571acd600eef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:32 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:44526
I20260806 12:23:32.146869 779 tablet_replica.cc:330] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe: stopping tablet replica
I20260806 12:23:32.147038 780 tablet_replica.cc:330] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f: stopping tablet replica
I20260806 12:23:32.147071 779 raft_consensus.cc:2229] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:32.147163 780 raft_consensus.cc:2229] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:32.147202 779 raft_consensus.cc:2258] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:32.147277 780 raft_consensus.cc:2258] T fd3989e971f34164a9e4571acd600eef P 1a718cad601d4fda828474f2d5249b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:32.147450 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 32327
I20260806 12:23:32.161809 782 tablet_replica.cc:330] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1: stopping tablet replica
I20260806 12:23:32.162039 782 raft_consensus.cc:2229] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:23:32.162565 782 raft_consensus.cc:2258] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:32.163455 782 ts_tablet_manager.cc:1905] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:32.165486 779 ts_tablet_manager.cc:1905] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:32.166851 782 ts_tablet_manager.cc:1918] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 12:23:32.166967 782 log.cc:1196] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/fd3989e971f34164a9e4571acd600eef
I20260806 12:23:32.167268 782 ts_tablet_manager.cc:1939] T fd3989e971f34164a9e4571acd600eef P 21523712491c4b759bc7e48ee84123e1: Deleting consensus metadata
I20260806 12:23:32.168560 779 ts_tablet_manager.cc:1918] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 12:23:32.168668 779 log.cc:1196] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/fd3989e971f34164a9e4571acd600eef
I20260806 12:23:32.169025 779 ts_tablet_manager.cc:1939] T fd3989e971f34164a9e4571acd600eef P 9f9eafc562da4dfea97fe03db90f27fe: Deleting consensus metadata
I20260806 12:23:32.169411 32193 catalog_manager.cc:4841] TS 21523712491c4b759bc7e48ee84123e1 (127.28.210.66:35587): tablet fd3989e971f34164a9e4571acd600eef (table test-table [id=90562c067d2949a7bc7b4613883a4c6d]) successfully deleted
I20260806 12:23:32.169993 32197 catalog_manager.cc:4841] TS 9f9eafc562da4dfea97fe03db90f27fe (127.28.210.67:41935): tablet fd3989e971f34164a9e4571acd600eef (table test-table [id=90562c067d2949a7bc7b4613883a4c6d]) successfully deleted
W20260806 12:23:32.173312 32193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:33339 (error 108)
I20260806 12:23:32.173568 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 32553
I20260806 12:23:32.183633 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:32.180636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:33339 (local address 127.0.0.1:60384)
0806 12:23:32.183314 (+ 2678us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:32.183475 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2553,"thread_start_us":2399,"threads_started":1}
W20260806 12:23:32.186012 32193 catalog_manager.cc:4584] TS 1a718cad601d4fda828474f2d5249b1f (127.28.210.65:33339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd3989e971f34164a9e4571acd600eef: Network error: Client connection negotiation failed: client connection to 127.28.210.65:33339: connect: Connection refused (error 111)
I20260806 12:23:32.199505 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 335
I20260806 12:23:32.224417 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 32171
2026-08-06T12:23:32Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7054 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:23:32 dist-test-slave-lgr5 krb5kdc[793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:23:32 dist-test-slave-lgr5 krb5kdc[793](info): set up 2 sockets
Aug 06 12:23:32 dist-test-slave-lgr5 krb5kdc[793](info): commencing operation
krb5kdc: starting...
W20260806 12:23:33.518059 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260806 12:23:33.747681 29513 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.229s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260806 12:23:33.946014 29513 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.198s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260806 12:23:34.082880 29513 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.137s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/test-user.keytab.
W20260806 12:23:34.505445 29513 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.422s user 0.000s sys 0.002s
Aug 06 12:23:34 dist-test-slave-lgr5 krb5kdc[793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:23:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
W20260806 12:23:35.053494 29513 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.210.126: real 0.514s user 0.000s sys 0.005s
I20260806 12:23:35.054248 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:42213
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:42213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:35.191414 812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:35.191735 812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:35.191779 812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:35.196005 812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:35.196122 812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:35.196143 812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:35.196158 812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:35.196172 812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:35.201870 812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:42213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:42213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:35.203212 812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:35.204397 812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:35.215062 824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:35.215368 822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:35.215186 818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:35.215616 812 server_base.cc:1034] running on GCE node
I20260806 12:23:35.216645 812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:35.218300 812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:35.219602 812 hybrid_clock.cc:648] HybridClock initialized: now 1786019015219569 us; error 59 us; skew 500 ppm
Aug 06 12:23:35 dist-test-slave-lgr5 krb5kdc[793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:35.223884 812 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:23:35.225615 812 webserver.cc:460] Webserver started at http://127.28.210.126:46021/ using document root <none> and password file <none>
I20260806 12:23:35.226518 812 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:35.226615 812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:35.226922 812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:35.230803 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "c8390b1b67794bdcb13f9b828cb650ce"
format_stamp: "Formatted at 2026-08-06 12:23:35 on dist-test-slave-lgr5"
server_key: "9c1e31c85224c3628ff8bf66e1f6eb46"
server_key_iv: "9a5b782edb2ff2b4dd1835998a209e75"
server_key_version: "encryptionkey@0"
I20260806 12:23:35.231695 812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "c8390b1b67794bdcb13f9b828cb650ce"
format_stamp: "Formatted at 2026-08-06 12:23:35 on dist-test-slave-lgr5"
server_key: "9c1e31c85224c3628ff8bf66e1f6eb46"
server_key_iv: "9a5b782edb2ff2b4dd1835998a209e75"
server_key_version: "encryptionkey@0"
I20260806 12:23:35.236781 812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 12:23:35.240463 835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:35.242472 812 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:23:35.242713 812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "c8390b1b67794bdcb13f9b828cb650ce"
format_stamp: "Formatted at 2026-08-06 12:23:35 on dist-test-slave-lgr5"
server_key: "9c1e31c85224c3628ff8bf66e1f6eb46"
server_key_iv: "9a5b782edb2ff2b4dd1835998a209e75"
server_key_version: "encryptionkey@0"
I20260806 12:23:35.242899 812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:35.262256 812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:35.266551 812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:35.266778 812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:35.275085 812 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:42213
I20260806 12:23:35.275121 915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:42213 every 8 connection(s)
I20260806 12:23:35.276252 812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:35.279655 917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:35.284416 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 812
I20260806 12:23:35.284571 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:35.284945 29513 external_mini_cluster.cc:1428] Setting key b6341be2780ee948a5d2954ccbdcc16c
I20260806 12:23:35.289426 917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce: Bootstrap starting.
I20260806 12:23:35.292984 917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:35.294224 917 log.cc:824] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce: Log is configured to *not* fsync() on all Append() calls
Aug 06 12:23:35 dist-test-slave-lgr5 krb5kdc[793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:35.298182 917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce: No bootstrap required, opened a new log
I20260806 12:23:35.304433 917 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8390b1b67794bdcb13f9b828cb650ce" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42213 } }
I20260806 12:23:35.304878 917 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:35.304948 917 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8390b1b67794bdcb13f9b828cb650ce, State: Initialized, Role: FOLLOWER
I20260806 12:23:35.305564 917 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8390b1b67794bdcb13f9b828cb650ce" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42213 } }
I20260806 12:23:35.305756 917 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:35.305830 917 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:35.305935 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:35.287095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.28.210.126:42213)
0806 12:23:35.287782 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:35.287805 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:35.287859 (+ 54us) server_negotiation.cc:407] Connection header received
0806 12:23:35.288852 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:35.288895 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:35.289395 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:35.289832 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:35.290700 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:35.292416 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:35.293601 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:35.294479 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:35.294678 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:35.298385 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:35.298394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:35.298417 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:35.298462 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:35.301092 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:35.301761 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:35.301769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:35.301778 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:35.301980 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:35.302283 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:35.302286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:35.302288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:35.303222 (+ 934us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:35.303628 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:35.304237 (+ 609us) server_negotiation.cc:299] Negotiation successful
0806 12:23:35.304685 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":248,"threads_started":1}
I20260806 12:23:35.305948 917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:35.307418 917 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8390b1b67794bdcb13f9b828cb650ce" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42213 } }
I20260806 12:23:35.307852 917 leader_election.cc:302] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8390b1b67794bdcb13f9b828cb650ce; no voters:
I20260806 12:23:35.308295 917 leader_election.cc:288] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:35.308393 927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:35.308601 927 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [term 1 LEADER]: Becoming Leader. State: Replica: c8390b1b67794bdcb13f9b828cb650ce, State: Running, Role: LEADER
I20260806 12:23:35.309031 927 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8390b1b67794bdcb13f9b828cb650ce" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42213 } }
I20260806 12:23:35.309875 917 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:35.311332 930 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8390b1b67794bdcb13f9b828cb650ce. Latest consensus state: current_term: 1 leader_uuid: "c8390b1b67794bdcb13f9b828cb650ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8390b1b67794bdcb13f9b828cb650ce" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42213 } } }
I20260806 12:23:35.311522 930 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:35.311858 928 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8390b1b67794bdcb13f9b828cb650ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8390b1b67794bdcb13f9b828cb650ce" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42213 } } }
I20260806 12:23:35.311985 928 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:35.312019 938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:35.316622 938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:35.324554 938 catalog_manager.cc:1347] Generated new cluster ID: ae0e3f32d0aa4150be4e73956b81aee4
I20260806 12:23:35.324672 938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:35.338778 938 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:35.341657 938 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:35.358779 938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8390b1b67794bdcb13f9b828cb650ce: Generated new TSK 0
I20260806 12:23:35.359978 938 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:23:35.404204 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42213
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:35.543651 954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:35.544059 954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:35.544135 954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:35.548765 954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:35.548869 954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:35.548938 954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:23:35.555688 954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:35.557111 954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:35.558326 954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:35.578815 962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:35.578990 965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:35.578815 960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:35.579218 954 server_base.cc:1034] running on GCE node
I20260806 12:23:35.579833 954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:35.580857 954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:35.584391 954 hybrid_clock.cc:648] HybridClock initialized: now 1786019015584304 us; error 65 us; skew 500 ppm
Aug 06 12:23:35 dist-test-slave-lgr5 krb5kdc[793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:35.589010 954 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:23:35.590477 954 webserver.cc:460] Webserver started at http://127.28.210.65:43941/ using document root <none> and password file <none>
I20260806 12:23:35.591553 954 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:35.591713 954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:35.592020 954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:35.595528 954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "ae9e48a798d44fed8fdbd3e9c4bbd8ff"
format_stamp: "Formatted at 2026-08-06 12:23:35 on dist-test-slave-lgr5"
server_key: "86ca85faec88100645f0367371057ff5"
server_key_iv: "22adc10b94586f6a21a99e81515d003d"
server_key_version: "encryptionkey@0"
I20260806 12:23:35.596467 954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "ae9e48a798d44fed8fdbd3e9c4bbd8ff"
format_stamp: "Formatted at 2026-08-06 12:23:35 on dist-test-slave-lgr5"
server_key: "86ca85faec88100645f0367371057ff5"
server_key_iv: "22adc10b94586f6a21a99e81515d003d"
server_key_version: "encryptionkey@0"
I20260806 12:23:35.602962 954 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 12:23:35.607324 974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:35.609525 954 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 12:23:35.609707 954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "ae9e48a798d44fed8fdbd3e9c4bbd8ff"
format_stamp: "Formatted at 2026-08-06 12:23:35 on dist-test-slave-lgr5"
server_key: "86ca85faec88100645f0367371057ff5"
server_key_iv: "22adc10b94586f6a21a99e81515d003d"
server_key_version: "encryptionkey@0"
I20260806 12:23:35.609886 954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:35.643556 954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:35.647298 954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:35.647478 954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:35.648171 954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:35.649272 954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:35.649329 954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:35.649374 954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:35.649395 954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:35.659291 954 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:36725
I20260806 12:23:35.659402 1173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:36725 every 8 connection(s)
I20260806 12:23:35.660504 954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:23:35.670164 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 954
I20260806 12:23:35.670317 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:23:35.670667 29513 external_mini_cluster.cc:1428] Setting key ace0afd0c6a23a2c6fda1c595b2f55df
Aug 06 12:23:35 dist-test-slave-lgr5 krb5kdc[793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:35.679716 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:35.662979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:43269 (local address 127.28.210.126:42213)
0806 12:23:35.663159 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:35.663163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:35.664183 (+ 1020us) server_negotiation.cc:407] Connection header received
0806 12:23:35.665792 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:35.665796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:35.665858 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:35.665951 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:35.667442 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:35.668023 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:35.669496 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:35.669812 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:35.669939 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:35.673809 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:35.673814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:35.673816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:35.673846 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:35.677046 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:35.677822 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:35.677826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:35.677827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:35.677938 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:35.678363 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:35.678366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:35.678367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:35.678598 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:35.678695 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:35.679344 (+ 649us) server_negotiation.cc:299] Negotiation successful
0806 12:23:35.679519 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 12:23:35.680896 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:35.663301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:42213 (local address 127.28.210.65:43269)
0806 12:23:35.663974 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:35.664030 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:35.665510 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:35.666212 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:35.666220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:35.666659 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:35.667249 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:35.667265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:35.668209 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:35.668213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:35.669291 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:35.669304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:35.670025 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:35.670029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:35.670256 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:35.671069 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:35.671095 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:35.673545 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:35.677289 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:35.677296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:35.677317 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:35.677694 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:35.678054 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:35.678057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:35.678059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:35.678253 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:35.678723 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:35.678731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:35.679067 (+ 336us) client_negotiation.cc:769] Sending connection context
0806 12:23:35.679277 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 12:23:35.679709 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":157,"threads_started":1}
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:23:35.682777 1174 heartbeater.cc:347] Connected to a master server at 127.28.210.126:42213
I20260806 12:23:35.683101 1174 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:35.683874 1174 heartbeater.cc:511] Master 127.28.210.126:42213 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:23:35.686823 858 ts_manager.cc:194] Registered new tserver with Master: ae9e48a798d44fed8fdbd3e9c4bbd8ff (127.28.210.65:36725)
I20260806 12:23:35.689065 858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:43269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:23:35.709441 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42213
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:35.844626 1182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:35.845103 1182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:35.845160 1182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:35.850708 1182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:35.850816 1182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:35.850883 1182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:23:35.857040 1182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:35.858342 1182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:35.859539 1182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:35.872138 1188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:35.872295 1190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:35.872262 1187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:35.872234 1182 server_base.cc:1034] running on GCE node
I20260806 12:23:35.873029 1182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:35.873843 1182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:35.875075 1182 hybrid_clock.cc:648] HybridClock initialized: now 1786019015875030 us; error 58 us; skew 500 ppm
Aug 06 12:23:35 dist-test-slave-lgr5 krb5kdc[793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:35.878535 1182 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:23:35.880240 1182 webserver.cc:460] Webserver started at http://127.28.210.66:39195/ using document root <none> and password file <none>
I20260806 12:23:35.881037 1182 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:35.881239 1182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:35.881562 1182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:35.884636 1182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "8c3041004f854d8fb05830a698aae13c"
format_stamp: "Formatted at 2026-08-06 12:23:35 on dist-test-slave-lgr5"
server_key: "d50cb218faf02ba7d258d4bc8a90ff24"
server_key_iv: "5466f5f4cc0ed1e8c49e570c26c0957a"
server_key_version: "encryptionkey@0"
I20260806 12:23:35.885445 1182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "8c3041004f854d8fb05830a698aae13c"
format_stamp: "Formatted at 2026-08-06 12:23:35 on dist-test-slave-lgr5"
server_key: "d50cb218faf02ba7d258d4bc8a90ff24"
server_key_iv: "5466f5f4cc0ed1e8c49e570c26c0957a"
server_key_version: "encryptionkey@0"
I20260806 12:23:35.892788 1182 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260806 12:23:35.896344 1202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:35.898470 1182 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:23:35.898670 1182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "8c3041004f854d8fb05830a698aae13c"
format_stamp: "Formatted at 2026-08-06 12:23:35 on dist-test-slave-lgr5"
server_key: "d50cb218faf02ba7d258d4bc8a90ff24"
server_key_iv: "5466f5f4cc0ed1e8c49e570c26c0957a"
server_key_version: "encryptionkey@0"
I20260806 12:23:35.898818 1182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:35.911690 1182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:35.915750 1182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:35.915966 1182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:35.916746 1182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:35.917753 1182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:35.917811 1182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:35.917874 1182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:35.917901 1182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:35.928766 1182 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:38449
I20260806 12:23:35.928828 1378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:38449 every 8 connection(s)
I20260806 12:23:35.930058 1182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:23:35.940217 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 1182
I20260806 12:23:35.940359 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:23:35.940680 29513 external_mini_cluster.cc:1428] Setting key ff269832d0da018df872fe96a0bad50e
Aug 06 12:23:35 dist-test-slave-lgr5 krb5kdc[793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:35.950879 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:35.932895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:56743 (local address 127.28.210.126:42213)
0806 12:23:35.933146 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:35.933151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:35.934451 (+ 1300us) server_negotiation.cc:407] Connection header received
0806 12:23:35.936384 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:35.936387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:35.936502 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:35.936621 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:35.938455 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:35.939087 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:35.940916 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:35.941187 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:35.941312 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:35.944753 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:35.944757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:35.944760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:35.944793 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:35.946383 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:35.947245 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:35.947248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:35.947250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:35.947364 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:35.947921 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:35.947927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:35.947929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:35.948304 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:35.948432 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:35.949803 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0806 12:23:35.950145 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260806 12:23:35.951709 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:35.933417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:42213 (local address 127.28.210.66:56743)
0806 12:23:35.934196 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:35.934265 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:35.935958 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:35.936943 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:35.936953 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:35.937476 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:35.938165 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:35.938186 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:35.939261 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:35.939265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:35.940737 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:35.940754 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:35.941398 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:35.941403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:35.941642 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:35.942404 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:35.942429 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:35.944445 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:35.946594 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:35.946603 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:35.946628 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:35.947090 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:35.947470 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:35.947474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:35.947476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:35.947723 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:35.948452 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:35.948478 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:35.948967 (+ 489us) client_negotiation.cc:769] Sending connection context
0806 12:23:35.949295 (+ 328us) client_negotiation.cc:240] Negotiation successful
0806 12:23:35.949926 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":198,"threads_started":1}
I20260806 12:23:35.953716 1379 heartbeater.cc:347] Connected to a master server at 127.28.210.126:42213
I20260806 12:23:35.954105 1379 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:35.954967 1379 heartbeater.cc:511] Master 127.28.210.126:42213 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
I20260806 12:23:35.956765 858 ts_manager.cc:194] Registered new tserver with Master: 8c3041004f854d8fb05830a698aae13c (127.28.210.66:38449)
I20260806 12:23:35.957608 858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:56743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:23:35.983608 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42213
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:36.129253 1385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:36.129621 1385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:36.129669 1385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:36.134853 1385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:36.134948 1385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:36.135035 1385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:23:36.141734 1385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:36.143009 1385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:36.144189 1385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:36.155591 1399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:36.155637 1392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:36.155633 1395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:36.156119 1385 server_base.cc:1034] running on GCE node
I20260806 12:23:36.156657 1385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:36.157482 1385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:36.158702 1385 hybrid_clock.cc:648] HybridClock initialized: now 1786019016158661 us; error 56 us; skew 500 ppm
Aug 06 12:23:36 dist-test-slave-lgr5 krb5kdc[793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019016, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:36.162442 1385 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:23:36.163682 1385 webserver.cc:460] Webserver started at http://127.28.210.67:38649/ using document root <none> and password file <none>
I20260806 12:23:36.164626 1385 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:36.164726 1385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:36.164981 1385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:36.167659 1385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "2a025c87f6ee46c893a190b9248a58c8"
format_stamp: "Formatted at 2026-08-06 12:23:36 on dist-test-slave-lgr5"
server_key: "3117621a1a51d2e5389340fea0bb441f"
server_key_iv: "d14924f8a845b095f9e1a06f11d1adb6"
server_key_version: "encryptionkey@0"
I20260806 12:23:36.168512 1385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "2a025c87f6ee46c893a190b9248a58c8"
format_stamp: "Formatted at 2026-08-06 12:23:36 on dist-test-slave-lgr5"
server_key: "3117621a1a51d2e5389340fea0bb441f"
server_key_iv: "d14924f8a845b095f9e1a06f11d1adb6"
server_key_version: "encryptionkey@0"
I20260806 12:23:36.173355 1385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 12:23:36.176398 1411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:36.177915 1385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:36.178076 1385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "2a025c87f6ee46c893a190b9248a58c8"
format_stamp: "Formatted at 2026-08-06 12:23:36 on dist-test-slave-lgr5"
server_key: "3117621a1a51d2e5389340fea0bb441f"
server_key_iv: "d14924f8a845b095f9e1a06f11d1adb6"
server_key_version: "encryptionkey@0"
I20260806 12:23:36.178203 1385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:36.209666 1385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:36.214110 1385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:36.214416 1385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:36.215214 1385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:36.216603 1385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:36.216671 1385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:36.216720 1385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:36.216734 1385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:36.229003 1385 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:43325
I20260806 12:23:36.229071 1600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:43325 every 8 connection(s)
I20260806 12:23:36.230139 1385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:23:36.235896 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 1385
I20260806 12:23:36.236056 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:23:36.236455 29513 external_mini_cluster.cc:1428] Setting key 1b3d4830307bf8cf12b96ad48a916e35
Aug 06 12:23:36 dist-test-slave-lgr5 krb5kdc[793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019016, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:36.247453 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:36.232635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:49681 (local address 127.28.210.126:42213)
0806 12:23:36.232832 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:36.232838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:36.234009 (+ 1171us) server_negotiation.cc:407] Connection header received
0806 12:23:36.235594 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:36.235598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:36.235666 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:36.235755 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:36.237773 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:36.238503 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:36.239997 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:36.240412 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:36.240537 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:36.243392 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:36.243396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:36.243398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:36.243449 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:36.244954 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:36.245571 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:36.245574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:36.245575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:36.245655 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:36.246118 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:36.246122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:36.246123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:36.246401 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:36.246500 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:36.247143 (+ 643us) server_negotiation.cc:299] Negotiation successful
0806 12:23:36.247309 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 12:23:36.248368 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:36.233143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:42213 (local address 127.28.210.67:49681)
0806 12:23:36.233772 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:36.233840 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:36.235275 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:36.236131 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:36.236148 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:36.236774 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:36.237568 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:36.237595 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:36.238673 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:36.238677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:36.239817 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:36.239830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:36.240625 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:36.240628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:36.240809 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:36.241527 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:36.241547 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:36.243181 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:36.245100 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:36.245107 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:36.245124 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:36.245467 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:36.245776 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:36.245779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:36.245780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:36.245964 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:36.246558 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:36.246564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:36.246866 (+ 302us) client_negotiation.cc:769] Sending connection context
0806 12:23:36.247054 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 12:23:36.247363 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":154,"threads_started":1}
I20260806 12:23:36.249735 1603 heartbeater.cc:347] Connected to a master server at 127.28.210.126:42213
I20260806 12:23:36.250036 1603 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:36.250653 1603 heartbeater.cc:511] Master 127.28.210.126:42213 requested a full tablet report, sending...
I20260806 12:23:36.251744 858 ts_manager.cc:194] Registered new tserver with Master: 2a025c87f6ee46c893a190b9248a58c8 (127.28.210.67:43325)
I20260806 12:23:36.252395 858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:49681
I20260806 12:23:36.263190 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 12:23:36.280576 923 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:23:36.278803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.28.210.126:42213)
0806 12:23:36.278958 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:36.278963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:36.279152 (+ 189us) server_negotiation.cc:407] Connection header received
0806 12:23:36.279291 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:36.279293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:36.279341 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:36.279411 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:36.280429 (+ 1018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57230: BlockingRecv error: recv got EOF from 127.0.0.1:57230 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:23:36.282028 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 954
I20260806 12:23:36.294286 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 1182
I20260806 12:23:36.351311 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 1385
I20260806 12:23:36.374451 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 812
2026-08-06T12:23:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4139 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:23:36 dist-test-slave-lgr5 krb5kdc[1622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:23:36 dist-test-slave-lgr5 krb5kdc[1622](info): set up 2 sockets
Aug 06 12:23:36 dist-test-slave-lgr5 krb5kdc[1622](info): commencing operation
krb5kdc: starting...
W20260806 12:23:37.568282 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:23:37 dist-test-slave-lgr5 krb5kdc[1622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:23:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:23:37.728498 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:43421
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:43421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:37.862120 1638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:37.862516 1638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:37.862565 1638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:37.867043 1638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:37.867136 1638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:37.867156 1638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:37.867172 1638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:37.867187 1638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:37.873128 1638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:43421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:43421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:37.874336 1638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:37.875478 1638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:37.885082 1648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:37.885073 1650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:37.885073 1647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:37.885500 1638 server_base.cc:1034] running on GCE node
I20260806 12:23:37.886143 1638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:37.887379 1638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:37.888557 1638 hybrid_clock.cc:648] HybridClock initialized: now 1786019017888541 us; error 31 us; skew 500 ppm
Aug 06 12:23:37 dist-test-slave-lgr5 krb5kdc[1622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:37.895807 1638 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:23:37.897118 1638 webserver.cc:460] Webserver started at http://127.28.210.126:36593/ using document root <none> and password file <none>
I20260806 12:23:37.897763 1638 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:37.897850 1638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:37.898125 1638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:37.900590 1638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "7ecbe180ae0b448db78f0aa690b06630"
format_stamp: "Formatted at 2026-08-06 12:23:37 on dist-test-slave-lgr5"
server_key: "b2e5e70c1135dd82f4f85441f57698c6"
server_key_iv: "916df2fe0925ce096aafb83392eae815"
server_key_version: "encryptionkey@0"
I20260806 12:23:37.901456 1638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "7ecbe180ae0b448db78f0aa690b06630"
format_stamp: "Formatted at 2026-08-06 12:23:37 on dist-test-slave-lgr5"
server_key: "b2e5e70c1135dd82f4f85441f57698c6"
server_key_iv: "916df2fe0925ce096aafb83392eae815"
server_key_version: "encryptionkey@0"
I20260806 12:23:37.907042 1638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 12:23:37.910288 1662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:37.911934 1638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:23:37.912158 1638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "7ecbe180ae0b448db78f0aa690b06630"
format_stamp: "Formatted at 2026-08-06 12:23:37 on dist-test-slave-lgr5"
server_key: "b2e5e70c1135dd82f4f85441f57698c6"
server_key_iv: "916df2fe0925ce096aafb83392eae815"
server_key_version: "encryptionkey@0"
I20260806 12:23:37.912323 1638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:37.934602 1638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:37.938709 1638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:37.938934 1638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:37.948029 1638 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:43421
I20260806 12:23:37.948678 1750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:43421 every 8 connection(s)
I20260806 12:23:37.949431 1638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:37.953425 1751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:37.959568 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 1638
I20260806 12:23:37.959733 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:37.960067 29513 external_mini_cluster.cc:1428] Setting key 98cfcd263b1ff7a8ded27e6bdf5cb2ec
I20260806 12:23:37.961050 1751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630: Bootstrap starting.
I20260806 12:23:37.965024 1751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:37.966156 1751 log.cc:824] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:37.969081 1751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630: No bootstrap required, opened a new log
Aug 06 12:23:37 dist-test-slave-lgr5 krb5kdc[1622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:37.976554 1751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecbe180ae0b448db78f0aa690b06630" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43421 } }
I20260806 12:23:37.977044 1751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:37.977129 1751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecbe180ae0b448db78f0aa690b06630, State: Initialized, Role: FOLLOWER
I20260806 12:23:37.977952 1751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecbe180ae0b448db78f0aa690b06630" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43421 } }
I20260806 12:23:37.978183 1751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:37.978272 1751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:37.978385 1751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:37.980285 1751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecbe180ae0b448db78f0aa690b06630" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43421 } }
I20260806 12:23:37.980857 1751 leader_election.cc:302] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ecbe180ae0b448db78f0aa690b06630; no voters:
I20260806 12:23:37.981307 1751 leader_election.cc:288] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:37.981581 1757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:37.981942 1757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [term 1 LEADER]: Becoming Leader. State: Replica: 7ecbe180ae0b448db78f0aa690b06630, State: Running, Role: LEADER
I20260806 12:23:37.982609 1757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecbe180ae0b448db78f0aa690b06630" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43421 } }
I20260806 12:23:37.983098 1751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:37.984362 1754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:37.961820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.28.210.126:43421)
0806 12:23:37.962653 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:37.962674 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:37.962732 (+ 58us) server_negotiation.cc:407] Connection header received
0806 12:23:37.964174 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:37.964216 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:37.964822 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:37.965327 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:37.966254 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:37.967607 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:37.968708 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:37.969459 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:37.969659 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:37.974761 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:37.974770 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:37.974926 (+ 156us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:37.975217 (+ 291us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:37.978942 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:37.980087 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:37.980126 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:37.980136 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:37.980355 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:37.980778 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:37.980781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:37.980783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:37.981515 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:37.982486 (+ 971us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:37.982660 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 12:23:37.983040 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":234,"threads_started":1}
I20260806 12:23:37.986507 1759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ecbe180ae0b448db78f0aa690b06630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecbe180ae0b448db78f0aa690b06630" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43421 } } }
I20260806 12:23:37.986553 1761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ecbe180ae0b448db78f0aa690b06630. Latest consensus state: current_term: 1 leader_uuid: "7ecbe180ae0b448db78f0aa690b06630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecbe180ae0b448db78f0aa690b06630" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43421 } } }
I20260806 12:23:37.986699 1761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:37.986699 1759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:37.988432 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 12:23:37.988613 1777 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:23:37.988694 1777 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:23:37.989635 1778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:37.990432 1778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:37.997928 1778 catalog_manager.cc:1347] Generated new cluster ID: d2b2b8056431457c877aa7fe20f93b46
I20260806 12:23:37.998049 1778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:38.009474 1778 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:38.011057 1778 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:38.015455 1754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:38.008623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.28.210.126:43421)
0806 12:23:38.008761 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:38.008766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:38.009803 (+ 1037us) server_negotiation.cc:407] Connection header received
0806 12:23:38.009986 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:38.009988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:38.010038 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:38.010112 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:38.010799 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:38.011613 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:38.012708 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:38.013011 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:38.013120 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:38.014121 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:38.014124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:38.014127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:38.014758 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:38.014806 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:38.014830 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:38.014948 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:38.015100 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 12:23:38.015273 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 12:23:38.021502 1778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ecbe180ae0b448db78f0aa690b06630: Generated new TSK 0
I20260806 12:23:38.022548 1778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:23:38.033250 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 1638
2026-08-06T12:23:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1664 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:23:38 dist-test-slave-lgr5 krb5kdc[1803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:23:38 dist-test-slave-lgr5 krb5kdc[1803](info): set up 2 sockets
Aug 06 12:23:38 dist-test-slave-lgr5 krb5kdc[1803](info): commencing operation
krb5kdc: starting...
W20260806 12:23:39.271947 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:23:39 dist-test-slave-lgr5 krb5kdc[1803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:23:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:23:39.420625 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:36291
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:36291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:39.554797 1819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:39.555220 1819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:39.555294 1819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:39.560115 1819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:39.560206 1819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:39.560235 1819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:39.560259 1819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:39.560281 1819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:39.566561 1819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:36291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:36291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:39.567950 1819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:39.569247 1819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:39.582430 1828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:39.582434 1836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:39.582458 1832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:39.582798 1819 server_base.cc:1034] running on GCE node
I20260806 12:23:39.583628 1819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:39.585458 1819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:39.586697 1819 hybrid_clock.cc:648] HybridClock initialized: now 1786019019586662 us; error 56 us; skew 500 ppm
Aug 06 12:23:39 dist-test-slave-lgr5 krb5kdc[1803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019019, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:39.590199 1819 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:23:39.591476 1819 webserver.cc:460] Webserver started at http://127.28.210.126:42821/ using document root <none> and password file <none>
I20260806 12:23:39.592196 1819 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:39.592294 1819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:39.592569 1819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:39.595541 1819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "d88ec721ba7749b8a1683e654654f3d4"
format_stamp: "Formatted at 2026-08-06 12:23:39 on dist-test-slave-lgr5"
server_key: "815b3f90db2cdc9862991dc21497539b"
server_key_iv: "e391a7af6a7f6b6062efb6f8cbcba371"
server_key_version: "encryptionkey@0"
I20260806 12:23:39.596223 1819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "d88ec721ba7749b8a1683e654654f3d4"
format_stamp: "Formatted at 2026-08-06 12:23:39 on dist-test-slave-lgr5"
server_key: "815b3f90db2cdc9862991dc21497539b"
server_key_iv: "e391a7af6a7f6b6062efb6f8cbcba371"
server_key_version: "encryptionkey@0"
I20260806 12:23:39.600674 1819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:23:39.603727 1845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:39.605449 1819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:39.605612 1819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "d88ec721ba7749b8a1683e654654f3d4"
format_stamp: "Formatted at 2026-08-06 12:23:39 on dist-test-slave-lgr5"
server_key: "815b3f90db2cdc9862991dc21497539b"
server_key_iv: "e391a7af6a7f6b6062efb6f8cbcba371"
server_key_version: "encryptionkey@0"
I20260806 12:23:39.605762 1819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:39.622325 1819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:39.626696 1819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:39.626919 1819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:39.636245 1819 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:36291
I20260806 12:23:39.636513 1937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:36291 every 8 connection(s)
I20260806 12:23:39.637791 1819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:39.640578 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 1819
I20260806 12:23:39.640734 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:39.641070 29513 external_mini_cluster.cc:1428] Setting key ab7115baf106f6b248b337e83ebd79b1
I20260806 12:23:39.641666 1938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:39.650854 1938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4: Bootstrap starting.
Aug 06 12:23:39 dist-test-slave-lgr5 krb5kdc[1803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:39.654285 1938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:39.655191 1938 log.cc:824] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:39.660019 1938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4: No bootstrap required, opened a new log
I20260806 12:23:39.660073 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:39.642723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.28.210.126:36291)
0806 12:23:39.643241 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:39.643258 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:39.643299 (+ 41us) server_negotiation.cc:407] Connection header received
0806 12:23:39.644052 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:39.644074 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:39.644517 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:39.644844 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:39.645964 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:39.647166 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:39.648724 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:39.649545 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:39.649743 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:39.653571 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:39.653578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:39.653597 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:39.653642 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:39.655824 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:39.656808 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:39.656815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:39.656821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:39.656998 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:39.657358 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:39.657361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:39.657376 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:39.658048 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:39.658335 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:39.658758 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 12:23:39.659172 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":179,"threads_started":1}
I20260806 12:23:39.664484 1938 raft_consensus.cc:348] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88ec721ba7749b8a1683e654654f3d4" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36291 } }
I20260806 12:23:39.664822 1938 raft_consensus.cc:374] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:39.664863 1938 raft_consensus.cc:729] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88ec721ba7749b8a1683e654654f3d4, State: Initialized, Role: FOLLOWER
I20260806 12:23:39.665376 1938 consensus_queue.cc:260] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88ec721ba7749b8a1683e654654f3d4" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36291 } }
I20260806 12:23:39.665532 1938 raft_consensus.cc:388] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:39.665588 1938 raft_consensus.cc:482] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:39.665697 1938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:39.666872 1938 raft_consensus.cc:504] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88ec721ba7749b8a1683e654654f3d4" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36291 } }
I20260806 12:23:39.667304 1938 leader_election.cc:302] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d88ec721ba7749b8a1683e654654f3d4; no voters:
I20260806 12:23:39.667699 1938 leader_election.cc:288] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:39.667923 1950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:39.668311 1950 raft_consensus.cc:686] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [term 1 LEADER]: Becoming Leader. State: Replica: d88ec721ba7749b8a1683e654654f3d4, State: Running, Role: LEADER
I20260806 12:23:39.668720 1950 consensus_queue.cc:237] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88ec721ba7749b8a1683e654654f3d4" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36291 } }
I20260806 12:23:39.669615 1938 sys_catalog.cc:564] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:39.671120 1955 sys_catalog.cc:455] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d88ec721ba7749b8a1683e654654f3d4. Latest consensus state: current_term: 1 leader_uuid: "d88ec721ba7749b8a1683e654654f3d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88ec721ba7749b8a1683e654654f3d4" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36291 } } }
I20260806 12:23:39.671159 1954 sys_catalog.cc:455] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d88ec721ba7749b8a1683e654654f3d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88ec721ba7749b8a1683e654654f3d4" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36291 } } }
I20260806 12:23:39.671295 1955 sys_catalog.cc:458] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:39.671324 1954 sys_catalog.cc:458] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:39.671909 1964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:39.676614 1964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:39.685590 1964 catalog_manager.cc:1347] Generated new cluster ID: 738e168c190d4c029209566379e4fe3c
I20260806 12:23:39.685700 1964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:39.703346 1964 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:39.704586 1964 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:39.715514 1964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d88ec721ba7749b8a1683e654654f3d4: Generated new TSK 0
I20260806 12:23:39.716419 1964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:23:39.756287 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36291
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:39.885174 1986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:39.885492 1986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:39.885540 1986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:39.889863 1986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:39.889951 1986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:39.890021 1986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:23:39.895780 1986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:39.897033 1986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:39.898115 1986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:39.908830 1991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:39.908948 1996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:39.908983 1993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:39.909122 1986 server_base.cc:1034] running on GCE node
I20260806 12:23:39.909593 1986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:39.910331 1986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:39.911527 1986 hybrid_clock.cc:648] HybridClock initialized: now 1786019019911491 us; error 47 us; skew 500 ppm
Aug 06 12:23:39 dist-test-slave-lgr5 krb5kdc[1803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019019, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:39.914825 1986 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:23:39.916193 1986 webserver.cc:460] Webserver started at http://127.28.210.65:43355/ using document root <none> and password file <none>
I20260806 12:23:39.916842 1986 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:39.916919 1986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:39.917135 1986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:39.919653 1986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "4f5b9e4f9ed74fc4b44bc0f06d0ad8b1"
format_stamp: "Formatted at 2026-08-06 12:23:39 on dist-test-slave-lgr5"
server_key: "97751dd45bc3c29a7dda9282416352b3"
server_key_iv: "a0290f2b85a47f75aa2e839bc6ddc64d"
server_key_version: "encryptionkey@0"
I20260806 12:23:39.920320 1986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "4f5b9e4f9ed74fc4b44bc0f06d0ad8b1"
format_stamp: "Formatted at 2026-08-06 12:23:39 on dist-test-slave-lgr5"
server_key: "97751dd45bc3c29a7dda9282416352b3"
server_key_iv: "a0290f2b85a47f75aa2e839bc6ddc64d"
server_key_version: "encryptionkey@0"
I20260806 12:23:39.924580 1986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:23:39.927386 2008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:39.929216 1986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:39.929386 1986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "4f5b9e4f9ed74fc4b44bc0f06d0ad8b1"
format_stamp: "Formatted at 2026-08-06 12:23:39 on dist-test-slave-lgr5"
server_key: "97751dd45bc3c29a7dda9282416352b3"
server_key_iv: "a0290f2b85a47f75aa2e839bc6ddc64d"
server_key_version: "encryptionkey@0"
I20260806 12:23:39.929525 1986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:39.942605 1986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:39.946643 1986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:39.946902 1986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:39.947707 1986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:39.949009 1986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:39.949090 1986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:39.949158 1986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:39.949186 1986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:39.961367 1986 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:40743
I20260806 12:23:39.961405 2190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:40743 every 8 connection(s)
I20260806 12:23:39.962443 1986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:23:39.965543 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 1986
I20260806 12:23:39.965685 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:23:39.965994 29513 external_mini_cluster.cc:1428] Setting key bd5f37fe71e9e8b057f0b8a86b497899
Aug 06 12:23:39 dist-test-slave-lgr5 krb5kdc[1803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019019, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:23:39.983259 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:39.964949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:35097 (local address 127.28.210.126:36291)
0806 12:23:39.965290 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:39.965296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:39.966333 (+ 1037us) server_negotiation.cc:407] Connection header received
0806 12:23:39.968144 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:39.968148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:39.968212 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:39.968309 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:39.970103 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:39.970771 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:39.972665 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:39.973069 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:39.973199 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:39.977241 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:39.977248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:39.977251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:39.977310 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:39.979433 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:39.980664 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:39.980670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:39.980672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:39.980878 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:39.981645 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:39.981652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:39.981653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:39.981989 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:39.982100 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:39.982879 (+ 779us) server_negotiation.cc:299] Negotiation successful
0806 12:23:39.983083 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 12:23:39.984427 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:39.965412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36291 (local address 127.28.210.65:35097)
0806 12:23:39.966098 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:39.966175 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:39.967836 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:39.968536 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:39.968545 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:39.968995 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:39.969870 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:39.969899 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:39.970974 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:39.970979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:39.972481 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:39.972497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:39.973351 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:39.973358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:39.973615 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:39.974402 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:39.974429 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:39.976902 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:39.979700 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:39.979711 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:39.979738 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:39.980405 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:39.981103 (+ 698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:39.981108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:39.981110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:39.981418 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:39.982127 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:39.982137 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:39.982521 (+ 384us) client_negotiation.cc:769] Sending connection context
0806 12:23:39.982737 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 12:23:39.983169 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":190,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:23:39.986500 2191 heartbeater.cc:347] Connected to a master server at 127.28.210.126:36291
I20260806 12:23:39.986869 2191 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:39.987618 2191 heartbeater.cc:511] Master 127.28.210.126:36291 requested a full tablet report, sending...
I20260806 12:23:39.989832 1873 ts_manager.cc:194] Registered new tserver with Master: 4f5b9e4f9ed74fc4b44bc0f06d0ad8b1 (127.28.210.65:40743)
I20260806 12:23:39.991484 1873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:35097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:23:40.014454 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36291
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:40.147002 2200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:40.147341 2200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:40.147408 2200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:40.151899 2200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:40.152052 2200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:40.152199 2200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:23:40.158645 2200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:40.160215 2200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:40.161482 2200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:40.172897 2208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:40.172945 2212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:40.173128 2200 server_base.cc:1034] running on GCE node
W20260806 12:23:40.173257 2210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:40.173672 2200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:40.174409 2200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:40.175590 2200 hybrid_clock.cc:648] HybridClock initialized: now 1786019020175568 us; error 36 us; skew 500 ppm
Aug 06 12:23:40 dist-test-slave-lgr5 krb5kdc[1803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:40.179026 2200 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:23:40.180197 2200 webserver.cc:460] Webserver started at http://127.28.210.66:34473/ using document root <none> and password file <none>
I20260806 12:23:40.180835 2200 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:40.180910 2200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:40.181128 2200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:40.183427 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "265fe1fb60df4324a941b5e1d02e4ca7"
format_stamp: "Formatted at 2026-08-06 12:23:40 on dist-test-slave-lgr5"
server_key: "a7afb8cc887bbdbfe1250a0d1c6e532c"
server_key_iv: "d1281cc634421a1fb6defde66b2425f4"
server_key_version: "encryptionkey@0"
I20260806 12:23:40.184031 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "265fe1fb60df4324a941b5e1d02e4ca7"
format_stamp: "Formatted at 2026-08-06 12:23:40 on dist-test-slave-lgr5"
server_key: "a7afb8cc887bbdbfe1250a0d1c6e532c"
server_key_iv: "d1281cc634421a1fb6defde66b2425f4"
server_key_version: "encryptionkey@0"
I20260806 12:23:40.188433 2200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:23:40.191282 2224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:40.192842 2200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:40.193003 2200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "265fe1fb60df4324a941b5e1d02e4ca7"
format_stamp: "Formatted at 2026-08-06 12:23:40 on dist-test-slave-lgr5"
server_key: "a7afb8cc887bbdbfe1250a0d1c6e532c"
server_key_iv: "d1281cc634421a1fb6defde66b2425f4"
server_key_version: "encryptionkey@0"
I20260806 12:23:40.193132 2200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:40.212222 2200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:40.216522 2200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:40.216784 2200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:40.217540 2200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:40.218819 2200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:40.218910 2200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:40.218967 2200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:40.218993 2200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:40.229681 2200 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:38811
I20260806 12:23:40.229743 2402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:38811 every 8 connection(s)
I20260806 12:23:40.230784 2200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:23:40.234290 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 2200
I20260806 12:23:40.234436 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:23:40.234756 29513 external_mini_cluster.cc:1428] Setting key 8d8592e6a2519795cb0f202736447906
Aug 06 12:23:40 dist-test-slave-lgr5 krb5kdc[1803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
I20260806 12:23:40.247856 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:40.233090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:36519 (local address 127.28.210.126:36291)
0806 12:23:40.233232 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:40.233237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:40.234215 (+ 978us) server_negotiation.cc:407] Connection header received
0806 12:23:40.236178 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:40.236182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:40.236244 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:40.236364 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:40.237812 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:40.238564 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:40.240461 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:40.240779 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:40.240905 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:40.243843 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:40.243846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:40.243848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:40.243880 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:40.245364 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:40.246058 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:40.246061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:40.246062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:40.246139 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:40.246595 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:40.246597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:40.246598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:40.246840 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:40.246927 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:40.247566 (+ 639us) server_negotiation.cc:299] Negotiation successful
0806 12:23:40.247695 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 12:23:40.248802 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:40.233473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36291 (local address 127.28.210.66:36519)
0806 12:23:40.234039 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:40.234092 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:40.235840 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:40.236580 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:40.236589 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:40.237039 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:40.237631 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:40.237647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:40.238739 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:40.238743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:40.240269 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:40.240286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:40.240991 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:40.240995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:40.241172 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:40.241814 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:40.241834 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:40.243580 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:40.245520 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:40.245527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:40.245549 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:40.245950 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:40.246222 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:40.246226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:40.246228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:40.246445 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:40.246954 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:40.246960 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:40.247265 (+ 305us) client_negotiation.cc:769] Sending connection context
0806 12:23:40.247457 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 12:23:40.247761 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":142,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
I20260806 12:23:40.250878 2403 heartbeater.cc:347] Connected to a master server at 127.28.210.126:36291
I20260806 12:23:40.251227 2403 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:40.252004 2403 heartbeater.cc:511] Master 127.28.210.126:36291 requested a full tablet report, sending...
I20260806 12:23:40.253723 1873 ts_manager.cc:194] Registered new tserver with Master: 265fe1fb60df4324a941b5e1d02e4ca7 (127.28.210.66:38811)
I20260806 12:23:40.254382 1873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:36519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:23:40.272835 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36291
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:40.407104 2415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:40.407454 2415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:40.407501 2415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:40.412122 2415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:40.412217 2415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:40.412298 2415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:23:40.419176 2415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:40.420779 2415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:40.422101 2415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:40.434953 2421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:40.434953 2425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:40.434924 2420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:40.435734 2415 server_base.cc:1034] running on GCE node
I20260806 12:23:40.436471 2415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:40.437405 2415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:40.438705 2415 hybrid_clock.cc:648] HybridClock initialized: now 1786019020438662 us; error 49 us; skew 500 ppm
Aug 06 12:23:40 dist-test-slave-lgr5 krb5kdc[1803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:40.443150 2415 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:23:40.444720 2415 webserver.cc:460] Webserver started at http://127.28.210.67:42061/ using document root <none> and password file <none>
I20260806 12:23:40.445400 2415 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:40.445485 2415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:40.445708 2415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:40.448421 2415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "ae255a8a44574466942a7507b5e42b51"
format_stamp: "Formatted at 2026-08-06 12:23:40 on dist-test-slave-lgr5"
server_key: "3f98d83eeb74c2040f98d5eab73bd8b3"
server_key_iv: "b25f3238c1cefa3c8dad93091270da7d"
server_key_version: "encryptionkey@0"
I20260806 12:23:40.449129 2415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "ae255a8a44574466942a7507b5e42b51"
format_stamp: "Formatted at 2026-08-06 12:23:40 on dist-test-slave-lgr5"
server_key: "3f98d83eeb74c2040f98d5eab73bd8b3"
server_key_iv: "b25f3238c1cefa3c8dad93091270da7d"
server_key_version: "encryptionkey@0"
I20260806 12:23:40.455065 2415 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 12:23:40.458547 2437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:40.460225 2415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:40.460395 2415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "ae255a8a44574466942a7507b5e42b51"
format_stamp: "Formatted at 2026-08-06 12:23:40 on dist-test-slave-lgr5"
server_key: "3f98d83eeb74c2040f98d5eab73bd8b3"
server_key_iv: "b25f3238c1cefa3c8dad93091270da7d"
server_key_version: "encryptionkey@0"
I20260806 12:23:40.460546 2415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:40.496496 2415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:40.501654 2415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:40.501850 2415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:40.502528 2415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:40.503605 2415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:40.503664 2415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:40.503710 2415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:40.503734 2415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:40.515030 2415 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:38585
I20260806 12:23:40.515053 2617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:38585 every 8 connection(s)
I20260806 12:23:40.516196 2415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:23:40.525298 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 2415
I20260806 12:23:40.525462 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:23:40.525815 29513 external_mini_cluster.cc:1428] Setting key 15b2f214c15ee82e25b2ffc09d11f299
Aug 06 12:23:40 dist-test-slave-lgr5 krb5kdc[1803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:40.531890 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:40.518337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:46547 (local address 127.28.210.126:36291)
0806 12:23:40.518489 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:40.518492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:40.519494 (+ 1002us) server_negotiation.cc:407] Connection header received
0806 12:23:40.521258 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:40.521263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:40.521325 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:40.521413 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:40.522867 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:40.523448 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:40.524802 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:40.525047 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:40.525151 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:40.528142 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:40.528146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:40.528149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:40.528189 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:40.529638 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:40.530199 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:40.530203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:40.530205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:40.530287 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:40.530639 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:40.530643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:40.530645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:40.530857 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:40.530956 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:40.531552 (+ 596us) server_negotiation.cc:299] Negotiation successful
0806 12:23:40.531699 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 12:23:40.532909 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:40.518703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36291 (local address 127.28.210.67:46547)
0806 12:23:40.519278 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:40.519326 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:40.520953 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:40.521627 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:40.521637 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:40.522087 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:40.522669 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:40.522686 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:40.523569 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:40.523572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:40.524652 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:40.524665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:40.525257 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:40.525262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:40.525511 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:40.526223 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:40.526243 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:40.527933 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:40.529768 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:40.529775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:40.529796 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:40.530096 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:40.530383 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:40.530386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:40.530388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:40.530544 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:40.530972 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:40.530980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:40.531278 (+ 298us) client_negotiation.cc:769] Sending connection context
0806 12:23:40.531487 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 12:23:40.531876 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":146,"threads_started":1}
I20260806 12:23:40.534265 2618 heartbeater.cc:347] Connected to a master server at 127.28.210.126:36291
I20260806 12:23:40.534569 2618 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:40.535189 2618 heartbeater.cc:511] Master 127.28.210.126:36291 requested a full tablet report, sending...
I20260806 12:23:40.536408 1873 ts_manager.cc:194] Registered new tserver with Master: ae255a8a44574466942a7507b5e42b51 (127.28.210.67:38585)
I20260806 12:23:40.537032 1873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:46547
I20260806 12:23:40.540343 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 12:23:40 dist-test-slave-lgr5 krb5kdc[1803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019020, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 12:23:40 dist-test-slave-lgr5 krb5kdc[1803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019020, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:40.568725 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:40.560021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.28.210.126:36291)
0806 12:23:40.560205 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:40.560210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:40.560350 (+ 140us) server_negotiation.cc:407] Connection header received
0806 12:23:40.560500 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:40.560502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:40.560548 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:40.560613 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:40.561373 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:40.561905 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:40.562723 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:40.562943 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:40.563032 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:40.565555 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:40.565558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:40.565560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:40.565589 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:40.566928 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:40.567366 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:40.567370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:40.567371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:40.567463 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:40.567717 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:40.567721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:40.567723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:40.567977 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:40.568131 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:40.568427 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 12:23:40.568561 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 12:23:40.569178 1873 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53216
I20260806 12:23:40.570158 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 1986
I20260806 12:23:40.586838 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 2200
I20260806 12:23:40.600862 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 2415
I20260806 12:23:40.625902 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 1819
2026-08-06T12:23:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2591 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:23:40 dist-test-slave-lgr5 krb5kdc[2638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:23:40 dist-test-slave-lgr5 krb5kdc[2638](info): set up 2 sockets
Aug 06 12:23:40 dist-test-slave-lgr5 krb5kdc[2638](info): commencing operation
krb5kdc: starting...
W20260806 12:23:41.863030 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:23:41 dist-test-slave-lgr5 krb5kdc[2638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:23:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:23:42.018123 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:35565
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:35565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:42.152706 2661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:42.153122 2661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:42.153199 2661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:42.157702 2661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:42.157819 2661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:42.157843 2661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:42.157863 2661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:42.157881 2661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:42.164206 2661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:35565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:35565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:42.165583 2661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:42.166889 2661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:23:42.177801 2661 server_base.cc:1034] running on GCE node
W20260806 12:23:42.177785 2669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:42.178032 2670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:42.177784 2673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:42.178753 2661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:42.180125 2661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:42.181344 2661 hybrid_clock.cc:648] HybridClock initialized: now 1786019022181327 us; error 39 us; skew 500 ppm
Aug 06 12:23:42 dist-test-slave-lgr5 krb5kdc[2638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:42.185154 2661 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:23:42.186528 2661 webserver.cc:460] Webserver started at http://127.28.210.126:33789/ using document root <none> and password file <none>
I20260806 12:23:42.187206 2661 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:42.187291 2661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:42.187561 2661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:42.189960 2661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "21c7d83b12c940d289d74ef77d5e6043"
format_stamp: "Formatted at 2026-08-06 12:23:42 on dist-test-slave-lgr5"
server_key: "28842d8d653d0ffa6760318a99b72af6"
server_key_iv: "4fe5298f3dce9e7085e0f52cc9b7f17c"
server_key_version: "encryptionkey@0"
I20260806 12:23:42.190586 2661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "21c7d83b12c940d289d74ef77d5e6043"
format_stamp: "Formatted at 2026-08-06 12:23:42 on dist-test-slave-lgr5"
server_key: "28842d8d653d0ffa6760318a99b72af6"
server_key_iv: "4fe5298f3dce9e7085e0f52cc9b7f17c"
server_key_version: "encryptionkey@0"
I20260806 12:23:42.194797 2661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 12:23:42.197613 2685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:42.198909 2661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 12:23:42.199048 2661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "21c7d83b12c940d289d74ef77d5e6043"
format_stamp: "Formatted at 2026-08-06 12:23:42 on dist-test-slave-lgr5"
server_key: "28842d8d653d0ffa6760318a99b72af6"
server_key_iv: "4fe5298f3dce9e7085e0f52cc9b7f17c"
server_key_version: "encryptionkey@0"
I20260806 12:23:42.199168 2661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:42.229722 2661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:42.233764 2661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:42.234000 2661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:42.244647 2661 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:35565
I20260806 12:23:42.244719 2773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:35565 every 8 connection(s)
I20260806 12:23:42.245958 2661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:42.247897 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 2661
I20260806 12:23:42.248044 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:42.248364 29513 external_mini_cluster.cc:1428] Setting key 02ae07a74f1725d04d4a1ba0b39d00dc
I20260806 12:23:42.249547 2776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:42.257115 2776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043: Bootstrap starting.
Aug 06 12:23:42 dist-test-slave-lgr5 krb5kdc[2638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:42.259848 2776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:42.260947 2776 log.cc:824] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:42.263620 2776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043: No bootstrap required, opened a new log
I20260806 12:23:42.264698 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:42.249906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.28.210.126:35565)
0806 12:23:42.250413 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:42.250467 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:42.250516 (+ 49us) server_negotiation.cc:407] Connection header received
0806 12:23:42.251426 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:42.251466 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:42.251911 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:42.252336 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:42.253095 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:42.254585 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:42.255530 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:42.256276 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:42.256451 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:42.259069 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:42.259076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:42.259094 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:42.259137 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:42.261253 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:42.261797 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:42.261804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:42.261810 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:42.261968 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:42.262338 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:42.262342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:42.262343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:42.262806 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:42.262974 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:42.263386 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 12:23:42.263764 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":142,"threads_started":1}
I20260806 12:23:42.267866 2776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c7d83b12c940d289d74ef77d5e6043" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35565 } }
I20260806 12:23:42.268193 2776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:42.268260 2776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c7d83b12c940d289d74ef77d5e6043, State: Initialized, Role: FOLLOWER
I20260806 12:23:42.268714 2776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c7d83b12c940d289d74ef77d5e6043" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35565 } }
I20260806 12:23:42.268860 2776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:42.268934 2776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:42.269038 2776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:42.270395 2776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c7d83b12c940d289d74ef77d5e6043" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35565 } }
I20260806 12:23:42.270819 2776 leader_election.cc:302] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21c7d83b12c940d289d74ef77d5e6043; no voters:
I20260806 12:23:42.271229 2776 leader_election.cc:288] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:42.271659 2784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:42.271991 2784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [term 1 LEADER]: Becoming Leader. State: Replica: 21c7d83b12c940d289d74ef77d5e6043, State: Running, Role: LEADER
I20260806 12:23:42.272375 2784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c7d83b12c940d289d74ef77d5e6043" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35565 } }
I20260806 12:23:42.272699 2776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:42.273986 2785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21c7d83b12c940d289d74ef77d5e6043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c7d83b12c940d289d74ef77d5e6043" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35565 } } }
I20260806 12:23:42.274134 2785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:42.274416 2786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21c7d83b12c940d289d74ef77d5e6043. Latest consensus state: current_term: 1 leader_uuid: "21c7d83b12c940d289d74ef77d5e6043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c7d83b12c940d289d74ef77d5e6043" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35565 } } }
I20260806 12:23:42.274528 2786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:42.274919 2798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:42.278052 2798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:42.284912 2798 catalog_manager.cc:1347] Generated new cluster ID: 75e3367fcfa0491da1dafe585fa40ead
I20260806 12:23:42.285035 2798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:42.297612 2798 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:42.298693 2798 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:42.306111 2798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21c7d83b12c940d289d74ef77d5e6043: Generated new TSK 0
I20260806 12:23:42.306926 2798 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:23:42.347754 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35565
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:42.476790 2814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:42.477200 2814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:42.477289 2814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:42.482506 2814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:42.482642 2814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:42.482745 2814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:23:42.489358 2814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:42.490871 2814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:42.492336 2814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:42.504860 2826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:42.504849 2832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:42.504849 2825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:42.505384 2814 server_base.cc:1034] running on GCE node
I20260806 12:23:42.506024 2814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:42.507097 2814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:42.508390 2814 hybrid_clock.cc:648] HybridClock initialized: now 1786019022508330 us; error 75 us; skew 500 ppm
Aug 06 12:23:42 dist-test-slave-lgr5 krb5kdc[2638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:42.511785 2814 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:23:42.513005 2814 webserver.cc:460] Webserver started at http://127.28.210.65:44411/ using document root <none> and password file <none>
I20260806 12:23:42.513670 2814 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:42.513746 2814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:42.513985 2814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:42.516543 2814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "750ec2c3d69d479e9ce6ad1cd3696e2c"
format_stamp: "Formatted at 2026-08-06 12:23:42 on dist-test-slave-lgr5"
server_key: "e84e5605c403524717c067b382b5dfcc"
server_key_iv: "5581e1870e224a035fdb715ebac1b24f"
server_key_version: "encryptionkey@0"
I20260806 12:23:42.517401 2814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "750ec2c3d69d479e9ce6ad1cd3696e2c"
format_stamp: "Formatted at 2026-08-06 12:23:42 on dist-test-slave-lgr5"
server_key: "e84e5605c403524717c067b382b5dfcc"
server_key_iv: "5581e1870e224a035fdb715ebac1b24f"
server_key_version: "encryptionkey@0"
I20260806 12:23:42.522877 2814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 12:23:42.526163 2843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:42.527683 2814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:23:42.527829 2814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "750ec2c3d69d479e9ce6ad1cd3696e2c"
format_stamp: "Formatted at 2026-08-06 12:23:42 on dist-test-slave-lgr5"
server_key: "e84e5605c403524717c067b382b5dfcc"
server_key_iv: "5581e1870e224a035fdb715ebac1b24f"
server_key_version: "encryptionkey@0"
I20260806 12:23:42.527957 2814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:42.542011 2814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:42.545707 2814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:42.545895 2814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:42.546650 2814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:42.547642 2814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:42.547796 2814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:42.547873 2814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:42.547906 2814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:42.558528 2814 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:33409
I20260806 12:23:42.558552 3013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:33409 every 8 connection(s)
I20260806 12:23:42.559628 2814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:23:42.567341 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 2814
I20260806 12:23:42.567471 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:23:42.567762 29513 external_mini_cluster.cc:1428] Setting key c2647c2fee29786d3dea4d99a89ff5e6
Aug 06 12:23:42 dist-test-slave-lgr5 krb5kdc[2638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:42.577955 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:42.561971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:35861 (local address 127.28.210.126:35565)
0806 12:23:42.562120 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:42.562124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:42.563165 (+ 1041us) server_negotiation.cc:407] Connection header received
0806 12:23:42.564767 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:42.564771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:42.564852 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:42.564928 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:42.566323 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:42.566883 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:42.568215 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:42.568500 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:42.568609 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:42.571842 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:42.571848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:42.571851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:42.571887 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:42.574101 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:42.574885 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:42.574889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:42.574890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:42.574998 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:42.576022 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:42.576027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:42.576028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:42.576436 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:42.576624 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:42.577499 (+ 875us) server_negotiation.cc:299] Negotiation successful
0806 12:23:42.577674 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 12:23:42.579015 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:42.562348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35565 (local address 127.28.210.65:35861)
0806 12:23:42.562972 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:42.563029 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:42.564516 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:42.565150 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:42.565158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:42.565602 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:42.566149 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:42.566164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:42.567021 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:42.567024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:42.568063 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:42.568075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:42.568693 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:42.568697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:42.569085 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:42.569832 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:42.569851 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:42.571536 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:42.574251 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:42.574259 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:42.574282 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:42.574711 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:42.575245 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:42.575249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:42.575251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:42.575687 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:42.576639 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:42.576650 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:42.577073 (+ 423us) client_negotiation.cc:769] Sending connection context
0806 12:23:42.577454 (+ 381us) client_negotiation.cc:240] Negotiation successful
0806 12:23:42.578014 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":141,"threads_started":1}
I20260806 12:23:42.580753 3014 heartbeater.cc:347] Connected to a master server at 127.28.210.126:35565
I20260806 12:23:42.581040 3014 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:23:42.581699 3014 heartbeater.cc:511] Master 127.28.210.126:35565 requested a full tablet report, sending...
I20260806 12:23:42.583585 2700 ts_manager.cc:194] Registered new tserver with Master: 750ec2c3d69d479e9ce6ad1cd3696e2c (127.28.210.65:33409)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:23:42.585105 2700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:35861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:23:42.603540 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35565
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:42.733668 3024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:42.734043 3024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:42.734117 3024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:42.738658 3024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:42.738755 3024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:42.738842 3024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:23:42.745071 3024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:42.746300 3024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:42.747511 3024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:42.757869 3031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:42.757877 3036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:42.757881 3034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:42.758121 3024 server_base.cc:1034] running on GCE node
I20260806 12:23:42.758580 3024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:42.759271 3024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:42.760456 3024 hybrid_clock.cc:648] HybridClock initialized: now 1786019022760427 us; error 42 us; skew 500 ppm
Aug 06 12:23:42 dist-test-slave-lgr5 krb5kdc[2638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:42.763502 3024 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:23:42.764626 3024 webserver.cc:460] Webserver started at http://127.28.210.66:42779/ using document root <none> and password file <none>
I20260806 12:23:42.765218 3024 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:42.765292 3024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:42.765532 3024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:42.767745 3024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "a6bb67e0a32249e3891ced5ecd6bd109"
format_stamp: "Formatted at 2026-08-06 12:23:42 on dist-test-slave-lgr5"
server_key: "ba0bdd4477e16d4257f0a52d032ae6c6"
server_key_iv: "f011fc887842b91b635b0f77f6eec267"
server_key_version: "encryptionkey@0"
I20260806 12:23:42.768379 3024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "a6bb67e0a32249e3891ced5ecd6bd109"
format_stamp: "Formatted at 2026-08-06 12:23:42 on dist-test-slave-lgr5"
server_key: "ba0bdd4477e16d4257f0a52d032ae6c6"
server_key_iv: "f011fc887842b91b635b0f77f6eec267"
server_key_version: "encryptionkey@0"
I20260806 12:23:42.772639 3024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 12:23:42.775817 3043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:42.777243 3024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:42.777390 3024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "a6bb67e0a32249e3891ced5ecd6bd109"
format_stamp: "Formatted at 2026-08-06 12:23:42 on dist-test-slave-lgr5"
server_key: "ba0bdd4477e16d4257f0a52d032ae6c6"
server_key_iv: "f011fc887842b91b635b0f77f6eec267"
server_key_version: "encryptionkey@0"
I20260806 12:23:42.777509 3024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:42.799353 3024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:42.802822 3024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:42.803025 3024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:42.803871 3024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:42.804963 3024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:42.805022 3024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:42.805083 3024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:42.805111 3024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:42.816190 3024 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:33893
I20260806 12:23:42.816247 3225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:33893 every 8 connection(s)
I20260806 12:23:42.817310 3024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:23:42.822983 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 3024
I20260806 12:23:42.823117 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:23:42.823417 29513 external_mini_cluster.cc:1428] Setting key 9021f76e5dcb47687dda8f072900ccec
Aug 06 12:23:42 dist-test-slave-lgr5 krb5kdc[2638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:42.833429 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:42.819551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:60299 (local address 127.28.210.126:35565)
0806 12:23:42.819736 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:42.819740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:42.820759 (+ 1019us) server_negotiation.cc:407] Connection header received
0806 12:23:42.822271 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:42.822274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:42.822339 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:42.822417 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:42.823802 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:42.824445 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:42.825833 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:42.826155 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:42.826271 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:42.829220 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:42.829225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:42.829228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:42.829259 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:42.830832 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:42.831425 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:42.831430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:42.831432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:42.831540 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:42.831998 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:42.832002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:42.832004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:42.832296 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:42.832401 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:42.833074 (+ 673us) server_negotiation.cc:299] Negotiation successful
0806 12:23:42.833234 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 12:23:42.834390 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:42.819979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35565 (local address 127.28.210.66:60299)
0806 12:23:42.820534 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:42.820583 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:42.822030 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:42.822637 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:42.822645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:42.823039 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:42.823643 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:42.823660 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:42.824617 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:42.824621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:42.825697 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:42.825710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:42.826401 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:42.826405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:42.826594 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:42.827379 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:42.827402 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:42.828995 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:42.830987 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:42.830994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:42.831010 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:42.831313 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:42.831667 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:42.831670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:42.831672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:42.831892 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:42.832425 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:42.832432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:42.832763 (+ 331us) client_negotiation.cc:769] Sending connection context
0806 12:23:42.832988 (+ 225us) client_negotiation.cc:240] Negotiation successful
0806 12:23:42.833337 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":150,"threads_started":1}
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
I20260806 12:23:42.835918 3230 heartbeater.cc:347] Connected to a master server at 127.28.210.126:35565
I20260806 12:23:42.836418 3230 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:42.837262 3230 heartbeater.cc:511] Master 127.28.210.126:35565 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
I20260806 12:23:42.838917 2700 ts_manager.cc:194] Registered new tserver with Master: a6bb67e0a32249e3891ced5ecd6bd109 (127.28.210.66:33893)
I20260806 12:23:42.839574 2700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:60299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:23:42.856380 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35565
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:42.989058 3236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:42.989435 3236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:42.989492 3236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:42.997602 3236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:42.997867 3236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:42.998026 3236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:23:43.005316 3236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:43.006744 3236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:43.007946 3236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:43.020773 3245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:43.020872 3236 server_base.cc:1034] running on GCE node
W20260806 12:23:43.020773 3247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:43.020804 3251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:43.021688 3236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:43.022552 3236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:43.023788 3236 hybrid_clock.cc:648] HybridClock initialized: now 1786019023023754 us; error 58 us; skew 500 ppm
Aug 06 12:23:43 dist-test-slave-lgr5 krb5kdc[2638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:43.031404 3236 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:23:43.032742 3236 webserver.cc:460] Webserver started at http://127.28.210.67:35565/ using document root <none> and password file <none>
I20260806 12:23:43.033488 3236 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:43.033596 3236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:43.033864 3236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:43.037842 3236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "32df45ed2266494a81e2349c59e2c83a"
format_stamp: "Formatted at 2026-08-06 12:23:43 on dist-test-slave-lgr5"
server_key: "a409304756dfdebf8af7ef40087dfffc"
server_key_iv: "e11e1e06909fc5bb3a432c45286eaa58"
server_key_version: "encryptionkey@0"
I20260806 12:23:43.038650 3236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "32df45ed2266494a81e2349c59e2c83a"
format_stamp: "Formatted at 2026-08-06 12:23:43 on dist-test-slave-lgr5"
server_key: "a409304756dfdebf8af7ef40087dfffc"
server_key_iv: "e11e1e06909fc5bb3a432c45286eaa58"
server_key_version: "encryptionkey@0"
I20260806 12:23:43.043691 3236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 12:23:43.047029 3265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:43.048913 3236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 12:23:43.049093 3236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "32df45ed2266494a81e2349c59e2c83a"
format_stamp: "Formatted at 2026-08-06 12:23:43 on dist-test-slave-lgr5"
server_key: "a409304756dfdebf8af7ef40087dfffc"
server_key_iv: "e11e1e06909fc5bb3a432c45286eaa58"
server_key_version: "encryptionkey@0"
I20260806 12:23:43.049244 3236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:43.098944 3236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:43.102936 3236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:43.103214 3236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:43.103995 3236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:43.105080 3236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:43.105140 3236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:43.105185 3236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:43.105199 3236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:43.119556 3236 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:36927
I20260806 12:23:43.119683 3455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:36927 every 8 connection(s)
I20260806 12:23:43.120656 3236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:23:43.127012 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 3236
I20260806 12:23:43.127157 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:23:43.127481 29513 external_mini_cluster.cc:1428] Setting key 8e231a6d7cf5f495a0ddc56a2257d5d6
Aug 06 12:23:43 dist-test-slave-lgr5 krb5kdc[2638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:43.136983 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:43.122820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:59005 (local address 127.28.210.126:35565)
0806 12:23:43.123062 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:43.123068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:43.124185 (+ 1117us) server_negotiation.cc:407] Connection header received
0806 12:23:43.125890 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:43.125893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:43.125975 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:43.126059 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:43.127621 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:43.128371 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:43.129686 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:43.129915 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:43.130017 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:43.133011 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:43.133015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:43.133017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:43.133046 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:43.134721 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:43.135322 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:43.135325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:43.135327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:43.135412 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:43.135760 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:43.135764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:43.135766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:43.135975 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:43.136068 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:43.136658 (+ 590us) server_negotiation.cc:299] Negotiation successful
0806 12:23:43.136804 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260806 12:23:43.138069 3459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:43.123345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35565 (local address 127.28.210.67:59005)
0806 12:23:43.123950 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:43.124006 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:43.125561 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:43.126279 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:43.126287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:43.126743 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:43.127352 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:43.127371 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:43.128527 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:43.128530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:43.129551 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:43.129563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:43.130069 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:43.130073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:43.130411 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:43.131174 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:43.131192 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:43.132798 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:43.134870 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:43.134879 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:43.134897 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:43.135206 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:43.135505 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:43.135508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:43.135510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:43.135664 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:43.136106 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:43.136114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:43.136406 (+ 292us) client_negotiation.cc:769] Sending connection context
0806 12:23:43.136601 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 12:23:43.136995 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":158,"threads_started":1}
I20260806 12:23:43.139647 3457 heartbeater.cc:347] Connected to a master server at 127.28.210.126:35565
I20260806 12:23:43.139943 3457 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:43.140635 3457 heartbeater.cc:511] Master 127.28.210.126:35565 requested a full tablet report, sending...
I20260806 12:23:43.141893 2700 ts_manager.cc:194] Registered new tserver with Master: 32df45ed2266494a81e2349c59e2c83a (127.28.210.67:36927)
I20260806 12:23:43.142588 2700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:59005
I20260806 12:23:43.154960 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 12:23:43 dist-test-slave-lgr5 krb5kdc[2638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 12:23:43 dist-test-slave-lgr5 krb5kdc[2638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.65@KRBTEST.COM
I20260806 12:23:43.187325 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:43.175509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.28.210.65:33409)
0806 12:23:43.176022 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:43.176033 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:43.176063 (+ 30us) server_negotiation.cc:407] Connection header received
0806 12:23:43.176161 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:43.176166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:43.176375 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:43.176513 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:43.177400 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:43.178232 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:43.179298 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:43.179753 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:43.179862 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:43.182964 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:43.182982 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:43.183002 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:43.183043 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:43.185158 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:43.185712 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:43.185719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:43.185720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:43.185826 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:43.186120 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:43.186123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:43.186124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:43.186454 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:43.186693 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:43.186942 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 12:23:43.187123 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":161,"threads_started":1}
Aug 06 12:23:43 dist-test-slave-lgr5 krb5kdc[2638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:43.199918 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:43.190626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.28.210.126:35565)
0806 12:23:43.190794 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:43.190798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:43.190813 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:23:43.190950 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:43.190952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:43.191000 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:43.191069 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:43.191720 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:43.192358 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:43.193227 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:43.193469 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:43.193568 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:43.196596 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:43.196601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:43.196603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:43.196639 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:43.198103 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:43.198645 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:43.198648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:43.198650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:43.198752 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:43.198983 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:43.198985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:43.198986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:43.199275 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:43.199396 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:43.199627 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 12:23:43.199750 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 12:23:43.200455 2700 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39338
I20260806 12:23:43.201180 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 2814
I20260806 12:23:43.216665 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 3024
I20260806 12:23:43.233414 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 3236
W20260806 12:23:43.262197 2766 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.28.210.126:35565
I20260806 12:23:43.287900 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 2661
2026-08-06T12:23:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2656 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:23:43 dist-test-slave-lgr5 krb5kdc[3471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:23:43 dist-test-slave-lgr5 krb5kdc[3471](info): set up 2 sockets
Aug 06 12:23:43 dist-test-slave-lgr5 krb5kdc[3471](info): commencing operation
krb5kdc: starting...
W20260806 12:23:44.509748 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:23:44 dist-test-slave-lgr5 krb5kdc[3471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:23:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:23:44.664898 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:35117
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:35117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:44.789366 3492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:44.789729 3492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:44.789778 3492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:44.794152 3492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:44.794230 3492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:44.794250 3492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:44.794266 3492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:44.794281 3492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:44.800330 3492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:35117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:35117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:44.801517 3492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:44.802574 3492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:44.811448 3504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:44.811432 3501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:44.811558 3492 server_base.cc:1034] running on GCE node
W20260806 12:23:44.811769 3502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:44.812448 3492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:44.813750 3492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:44.814924 3492 hybrid_clock.cc:648] HybridClock initialized: now 1786019024814912 us; error 38 us; skew 500 ppm
Aug 06 12:23:44 dist-test-slave-lgr5 krb5kdc[3471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019024, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:44.818081 3492 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:23:44.818928 3492 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:44.819006 3492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:44.819231 3492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:44.821537 3492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "b643d873a49e44bea949099c2a3b7bac"
format_stamp: "Formatted at 2026-08-06 12:23:44 on dist-test-slave-lgr5"
server_key: "9c1f4ccfcbf549f7b68996c52028ed6e"
server_key_iv: "23d1fc634b7408b87807a8bd888aa58e"
server_key_version: "encryptionkey@0"
I20260806 12:23:44.822139 3492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "b643d873a49e44bea949099c2a3b7bac"
format_stamp: "Formatted at 2026-08-06 12:23:44 on dist-test-slave-lgr5"
server_key: "9c1f4ccfcbf549f7b68996c52028ed6e"
server_key_iv: "23d1fc634b7408b87807a8bd888aa58e"
server_key_version: "encryptionkey@0"
I20260806 12:23:44.826335 3492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:23:44.829211 3516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:44.830680 3492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 12:23:44.830835 3492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "b643d873a49e44bea949099c2a3b7bac"
format_stamp: "Formatted at 2026-08-06 12:23:44 on dist-test-slave-lgr5"
server_key: "9c1f4ccfcbf549f7b68996c52028ed6e"
server_key_iv: "23d1fc634b7408b87807a8bd888aa58e"
server_key_version: "encryptionkey@0"
I20260806 12:23:44.830957 3492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:44.877064 3492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:44.880795 3492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:44.881039 3492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:44.888972 3492 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:35117
I20260806 12:23:44.889246 3598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:35117 every 8 connection(s)
I20260806 12:23:44.889773 3492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:44.893137 3601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:44.894830 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 3492
I20260806 12:23:44.894981 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:44.895292 29513 external_mini_cluster.cc:1428] Setting key b63566e5e1df63dd9ca3bcef0a02c744
I20260806 12:23:44.900527 3601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac: Bootstrap starting.
I20260806 12:23:44.903071 3601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:44.903877 3601 log.cc:824] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac: Log is configured to *not* fsync() on all Append() calls
Aug 06 12:23:44 dist-test-slave-lgr5 krb5kdc[3471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:44.906183 3601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac: No bootstrap required, opened a new log
I20260806 12:23:44.910255 3604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:44.896875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.28.210.126:35117)
0806 12:23:44.897407 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:44.897424 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:44.897548 (+ 124us) server_negotiation.cc:407] Connection header received
0806 12:23:44.898310 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:44.898329 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:44.898708 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:44.899093 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:44.899950 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:44.901053 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:44.901931 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:44.902436 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:44.902610 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:44.905309 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:44.905316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:44.905331 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:44.905361 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:44.907084 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:44.907541 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:44.907546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:44.907551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:44.907668 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:44.907950 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:44.907953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:44.907954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:44.908419 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:44.908641 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:44.908901 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 12:23:44.909215 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":145,"threads_started":1}
I20260806 12:23:44.910138 3601 raft_consensus.cc:348] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b643d873a49e44bea949099c2a3b7bac" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35117 } }
I20260806 12:23:44.910409 3601 raft_consensus.cc:374] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:44.910476 3601 raft_consensus.cc:729] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b643d873a49e44bea949099c2a3b7bac, State: Initialized, Role: FOLLOWER
I20260806 12:23:44.910943 3601 consensus_queue.cc:260] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b643d873a49e44bea949099c2a3b7bac" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35117 } }
I20260806 12:23:44.911089 3601 raft_consensus.cc:388] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:44.911160 3601 raft_consensus.cc:482] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:44.911262 3601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:44.912333 3601 raft_consensus.cc:504] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b643d873a49e44bea949099c2a3b7bac" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35117 } }
I20260806 12:23:44.912680 3601 leader_election.cc:302] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b643d873a49e44bea949099c2a3b7bac; no voters:
I20260806 12:23:44.913065 3601 leader_election.cc:288] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:44.913184 3612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:44.913399 3612 raft_consensus.cc:686] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [term 1 LEADER]: Becoming Leader. State: Replica: b643d873a49e44bea949099c2a3b7bac, State: Running, Role: LEADER
I20260806 12:23:44.913729 3612 consensus_queue.cc:237] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b643d873a49e44bea949099c2a3b7bac" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35117 } }
I20260806 12:23:44.914371 3601 sys_catalog.cc:564] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:44.915477 3615 sys_catalog.cc:455] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader b643d873a49e44bea949099c2a3b7bac. Latest consensus state: current_term: 1 leader_uuid: "b643d873a49e44bea949099c2a3b7bac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b643d873a49e44bea949099c2a3b7bac" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35117 } } }
I20260806 12:23:44.915632 3615 sys_catalog.cc:458] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:44.916066 3632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:44.915477 3613 sys_catalog.cc:455] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b643d873a49e44bea949099c2a3b7bac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b643d873a49e44bea949099c2a3b7bac" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35117 } } }
I20260806 12:23:44.916355 3613 sys_catalog.cc:458] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:44.919108 3632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:44.925402 3632 catalog_manager.cc:1347] Generated new cluster ID: c55cebb6aa554da1aa16db5befd0e65a
I20260806 12:23:44.925487 3632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:44.938704 3632 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:44.939983 3632 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:44.948546 3632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac: Generated new TSK 0
I20260806 12:23:44.949846 3632 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:23:44.992884 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35117
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:45.132980 3648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:45.133400 3648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:45.133639 3648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:45.139292 3648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:45.139405 3648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:45.139478 3648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:23:45.145972 3648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:45.147541 3648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:45.148926 3648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:45.161705 3656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:45.161705 3654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:45.161955 3648 server_base.cc:1034] running on GCE node
W20260806 12:23:45.161705 3658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:45.162634 3648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:45.163466 3648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:45.164655 3648 hybrid_clock.cc:648] HybridClock initialized: now 1786019025164620 us; error 54 us; skew 500 ppm
Aug 06 12:23:45 dist-test-slave-lgr5 krb5kdc[3471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:45.168322 3648 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:23:45.169214 3648 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:45.169286 3648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:45.169528 3648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:45.171912 3648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "4d9a8c96dc4148e4be9db3656c598c7f"
format_stamp: "Formatted at 2026-08-06 12:23:45 on dist-test-slave-lgr5"
server_key: "17e8f780083e72bfe8f49e3edb610748"
server_key_iv: "aadf0471f5ec1bbbbe7563f11c62fc01"
server_key_version: "encryptionkey@0"
I20260806 12:23:45.172606 3648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "4d9a8c96dc4148e4be9db3656c598c7f"
format_stamp: "Formatted at 2026-08-06 12:23:45 on dist-test-slave-lgr5"
server_key: "17e8f780083e72bfe8f49e3edb610748"
server_key_iv: "aadf0471f5ec1bbbbe7563f11c62fc01"
server_key_version: "encryptionkey@0"
I20260806 12:23:45.177366 3648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260806 12:23:45.180418 3667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:45.182215 3648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 12:23:45.182365 3648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "4d9a8c96dc4148e4be9db3656c598c7f"
format_stamp: "Formatted at 2026-08-06 12:23:45 on dist-test-slave-lgr5"
server_key: "17e8f780083e72bfe8f49e3edb610748"
server_key_iv: "aadf0471f5ec1bbbbe7563f11c62fc01"
server_key_version: "encryptionkey@0"
I20260806 12:23:45.182489 3648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:45.199855 3648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:45.204483 3648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:45.204681 3648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:45.205356 3648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:45.206339 3648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:45.206394 3648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:45.206439 3648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:45.206452 3648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:45.218084 3648 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:36723
I20260806 12:23:45.218143 3859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:36723 every 8 connection(s)
I20260806 12:23:45.218817 3648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:23:45.223265 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 3648
I20260806 12:23:45.223423 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:23:45.223764 29513 external_mini_cluster.cc:1428] Setting key 3dc2ddaa22145895c2deb414f14b2d62
Aug 06 12:23:45 dist-test-slave-lgr5 krb5kdc[3471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:23:45.237160 3604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.221128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:49355 (local address 127.28.210.126:35117)
0806 12:23:45.221320 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:45.221324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:45.222324 (+ 1000us) server_negotiation.cc:407] Connection header received
0806 12:23:45.224169 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:45.224172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:45.224238 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:45.224339 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:45.226027 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.226770 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.228349 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.228703 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.228860 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.232401 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:45.232405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:45.232408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:45.232455 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:45.234128 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:45.234776 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:45.234779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:45.234780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:45.234869 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:45.235392 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:45.235395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:45.235396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:45.235653 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:45.235744 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:45.236745 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0806 12:23:45.236911 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 12:23:45.238251 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.221515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35117 (local address 127.28.210.65:49355)
0806 12:23:45.222104 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:45.222157 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:45.223755 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:45.224591 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:45.224599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:45.225151 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:45.225852 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:45.225871 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.226947 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.226950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:45.228133 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:45.228148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.228952 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.228956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:45.229334 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.230118 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:45.230140 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:45.232161 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:45.234316 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:45.234322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:45.234341 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:45.234646 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:45.235005 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:45.235008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:45.235010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:45.235251 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:45.235802 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:45.235810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:45.236260 (+ 450us) client_negotiation.cc:769] Sending connection context
0806 12:23:45.236693 (+ 433us) client_negotiation.cc:240] Negotiation successful
0806 12:23:45.237080 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:23:45.239981 3864 heartbeater.cc:347] Connected to a master server at 127.28.210.126:35117
I20260806 12:23:45.240334 3864 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:45.240909 3864 heartbeater.cc:511] Master 127.28.210.126:35117 requested a full tablet report, sending...
I20260806 12:23:45.242728 3544 ts_manager.cc:194] Registered new tserver with Master: 4d9a8c96dc4148e4be9db3656c598c7f (127.28.210.65:36723)
I20260806 12:23:45.243942 3544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:49355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:23:45.259820 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35117
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:45.387835 3869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:45.388191 3869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:45.388262 3869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:45.392503 3869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:45.392611 3869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:45.392701 3869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:23:45.398489 3869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:45.399760 3869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:45.400919 3869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:45.412020 3882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:45.412021 3876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:45.412071 3869 server_base.cc:1034] running on GCE node
W20260806 12:23:45.412021 3877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:45.412827 3869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:45.413604 3869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:45.414798 3869 hybrid_clock.cc:648] HybridClock initialized: now 1786019025414772 us; error 38 us; skew 500 ppm
Aug 06 12:23:45 dist-test-slave-lgr5 krb5kdc[3471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:45.418388 3869 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:23:45.419277 3869 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:45.419368 3869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:45.419628 3869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:45.422165 3869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "fbab83d2837f41739171956214b03079"
format_stamp: "Formatted at 2026-08-06 12:23:45 on dist-test-slave-lgr5"
server_key: "94e5b3fd80fe8a15661fe1f01e640d2f"
server_key_iv: "12766b61cfc36b8f1d5fd976eaf76630"
server_key_version: "encryptionkey@0"
I20260806 12:23:45.422859 3869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "fbab83d2837f41739171956214b03079"
format_stamp: "Formatted at 2026-08-06 12:23:45 on dist-test-slave-lgr5"
server_key: "94e5b3fd80fe8a15661fe1f01e640d2f"
server_key_iv: "12766b61cfc36b8f1d5fd976eaf76630"
server_key_version: "encryptionkey@0"
I20260806 12:23:45.427906 3869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 12:23:45.431236 3890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:45.432809 3869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:45.432981 3869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "fbab83d2837f41739171956214b03079"
format_stamp: "Formatted at 2026-08-06 12:23:45 on dist-test-slave-lgr5"
server_key: "94e5b3fd80fe8a15661fe1f01e640d2f"
server_key_iv: "12766b61cfc36b8f1d5fd976eaf76630"
server_key_version: "encryptionkey@0"
I20260806 12:23:45.433120 3869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:45.456171 3869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:45.460024 3869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:45.460295 3869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:45.461017 3869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:45.462035 3869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:45.462091 3869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:45.462152 3869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:45.462177 3869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:45.472494 3869 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:37401
I20260806 12:23:45.472512 4066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:37401 every 8 connection(s)
I20260806 12:23:45.473217 3869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:23:45.479264 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 3869
I20260806 12:23:45.479396 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:23:45.479722 29513 external_mini_cluster.cc:1428] Setting key becf99d7aad4a03f4c35cbda344e2705
Aug 06 12:23:45 dist-test-slave-lgr5 krb5kdc[3471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:45.488242 3604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.475195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:58107 (local address 127.28.210.126:35117)
0806 12:23:45.475346 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:45.475351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:45.476311 (+ 960us) server_negotiation.cc:407] Connection header received
0806 12:23:45.477707 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:45.477710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:45.477769 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:45.477851 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:45.479380 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.480191 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.481449 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.481695 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.481787 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.484528 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:45.484532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:45.484534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:45.484563 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:45.485926 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:45.486542 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:45.486547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:45.486549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:45.486635 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:45.486973 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:45.486977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:45.486978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:45.487186 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:45.487276 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:45.487885 (+ 609us) server_negotiation.cc:299] Negotiation successful
0806 12:23:45.488032 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 12:23:45.489198 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.475620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35117 (local address 127.28.210.66:58107)
0806 12:23:45.476149 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:45.476199 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:45.477490 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:45.478101 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:45.478109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:45.478573 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:45.479160 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:45.479191 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.480359 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.480362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:45.481322 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:45.481334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.481858 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.481862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:45.482049 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.482703 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:45.482722 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:45.484294 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:45.486059 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:45.486066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:45.486088 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:45.486379 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:45.486725 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:45.486728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:45.486730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:45.486882 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:45.487290 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:45.487298 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:45.487623 (+ 325us) client_negotiation.cc:769] Sending connection context
0806 12:23:45.487829 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 12:23:45.488251 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
I20260806 12:23:45.490420 4067 heartbeater.cc:347] Connected to a master server at 127.28.210.126:35117
I20260806 12:23:45.490677 4067 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:45.491185 4067 heartbeater.cc:511] Master 127.28.210.126:35117 requested a full tablet report, sending...
I20260806 12:23:45.492439 3544 ts_manager.cc:194] Registered new tserver with Master: fbab83d2837f41739171956214b03079 (127.28.210.66:37401)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
I20260806 12:23:45.493269 3544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:58107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:23:45.512516 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35117
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:45.661504 4077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:45.662041 4077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:45.662156 4077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:45.667708 4077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:45.667793 4077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:45.667883 4077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:23:45.675002 4077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:45.676514 4077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:45.677788 4077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:45.689894 4085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:45.689894 4089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:45.689934 4086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:45.690107 4077 server_base.cc:1034] running on GCE node
I20260806 12:23:45.690651 4077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:45.691432 4077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:45.692618 4077 hybrid_clock.cc:648] HybridClock initialized: now 1786019025692598 us; error 44 us; skew 500 ppm
Aug 06 12:23:45 dist-test-slave-lgr5 krb5kdc[3471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:45.696350 4077 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:23:45.697726 4077 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:45.697845 4077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:45.698101 4077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:45.700968 4077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "5ebcec7866684cd3bc143f4f090670c9"
format_stamp: "Formatted at 2026-08-06 12:23:45 on dist-test-slave-lgr5"
server_key: "be454a1ee4e75d88d62324f99033d4b3"
server_key_iv: "dca0413f4d571e11afdc7683e1806eff"
server_key_version: "encryptionkey@0"
I20260806 12:23:45.701675 4077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "5ebcec7866684cd3bc143f4f090670c9"
format_stamp: "Formatted at 2026-08-06 12:23:45 on dist-test-slave-lgr5"
server_key: "be454a1ee4e75d88d62324f99033d4b3"
server_key_iv: "dca0413f4d571e11afdc7683e1806eff"
server_key_version: "encryptionkey@0"
I20260806 12:23:45.705981 4077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:23:45.708853 4096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:45.710443 4077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:45.710604 4077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "5ebcec7866684cd3bc143f4f090670c9"
format_stamp: "Formatted at 2026-08-06 12:23:45 on dist-test-slave-lgr5"
server_key: "be454a1ee4e75d88d62324f99033d4b3"
server_key_iv: "dca0413f4d571e11afdc7683e1806eff"
server_key_version: "encryptionkey@0"
I20260806 12:23:45.710742 4077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:45.730883 4077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:45.734630 4077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:45.734862 4077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:45.735563 4077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:45.736595 4077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:45.736650 4077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:45.736711 4077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:45.736737 4077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:45.747370 4077 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:39093
I20260806 12:23:45.747393 4272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:39093 every 8 connection(s)
I20260806 12:23:45.748183 4077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:23:45.755821 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 4077
I20260806 12:23:45.755982 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:23:45.756546 29513 external_mini_cluster.cc:1428] Setting key 946f6034cecd77a2fc090ed3ba19fe99
Aug 06 12:23:45 dist-test-slave-lgr5 krb5kdc[3471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:45.764871 3604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.750372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:43661 (local address 127.28.210.126:35117)
0806 12:23:45.750598 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:45.750602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:45.751580 (+ 978us) server_negotiation.cc:407] Connection header received
0806 12:23:45.753172 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:45.753175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:45.753247 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:45.753339 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:45.754712 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.755372 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.757315 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.757607 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.757719 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.760727 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:45.760732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:45.760734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:45.760767 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:45.762222 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:45.762974 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:45.762978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:45.762979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:45.763086 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:45.763460 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:45.763463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:45.763465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:45.763695 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:45.763786 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:45.764578 (+ 792us) server_negotiation.cc:299] Negotiation successful
0806 12:23:45.764720 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 12:23:45.765843 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.750789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35117 (local address 127.28.210.67:43661)
0806 12:23:45.751386 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:45.751439 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:45.752908 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:45.753557 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:45.753566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:45.754002 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:45.754550 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:45.754566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.755554 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.755559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:45.757185 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:45.757202 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.757781 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.757784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:45.757995 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.758668 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:45.758698 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:45.760498 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:45.762384 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:45.762391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:45.762409 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:45.762818 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:45.763169 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:45.763171 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:45.763172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:45.763329 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:45.763833 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:45.763840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:45.764236 (+ 396us) client_negotiation.cc:769] Sending connection context
0806 12:23:45.764450 (+ 214us) client_negotiation.cc:240] Negotiation successful
0806 12:23:45.764787 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":149,"threads_started":1}
I20260806 12:23:45.767218 4273 heartbeater.cc:347] Connected to a master server at 127.28.210.126:35117
I20260806 12:23:45.767500 4273 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:45.768024 4273 heartbeater.cc:511] Master 127.28.210.126:35117 requested a full tablet report, sending...
I20260806 12:23:45.769346 3544 ts_manager.cc:194] Registered new tserver with Master: 5ebcec7866684cd3bc143f4f090670c9 (127.28.210.67:39093)
I20260806 12:23:45.770036 3544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:43661
I20260806 12:23:45.770905 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:23:45.782884 3604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.773084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.28.210.126:35117)
0806 12:23:45.773213 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:45.773217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:45.773353 (+ 136us) server_negotiation.cc:407] Connection header received
0806 12:23:45.773463 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:45.773465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:45.773509 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:45.773584 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:45.774350 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.774887 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.776038 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.776357 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.776452 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.778873 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:45.778878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:45.778880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:45.778920 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:45.780668 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:45.781244 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:45.781248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:45.781250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:45.781369 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:45.781767 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:45.781771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:45.781773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:45.782173 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:45.782295 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:45.782617 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 12:23:45.782745 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 12:23:45.786615 3544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:23:45.790467 3544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:45.808490 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.801813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:36723 (local address 127.0.0.1:50074)
0806 12:23:45.802326 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:45.802396 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:45.802577 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:45.803428 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:45.803434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:45.803461 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:45.803671 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:45.803681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.805205 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.805210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:45.806545 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:45.806556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.807781 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.807786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:45.807934 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.807993 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 12:23:45.808156 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 12:23:45.808316 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":166,"threads_started":1}
I20260806 12:23:45.808573 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.802382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.28.210.66:37401)
0806 12:23:45.802889 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:45.802901 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:45.803168 (+ 267us) server_negotiation.cc:407] Connection header received
0806 12:23:45.803397 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:45.803402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:45.803596 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:45.803738 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:45.804210 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.805257 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.806939 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.807704 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.807795 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.807935 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:45.808288 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 12:23:45.808405 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":202,"threads_started":1}
I20260806 12:23:45.808789 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.802982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.28.210.67:39093)
0806 12:23:45.803453 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:45.803465 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:45.803711 (+ 246us) server_negotiation.cc:407] Connection header received
0806 12:23:45.803932 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:45.803938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:45.804075 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:45.804220 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:45.804698 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.805885 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.807221 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.807847 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.808238 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.808394 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:45.808514 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 12:23:45.808620 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":109,"threads_started":1}
I20260806 12:23:45.808956 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.802090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:37401 (local address 127.0.0.1:43320)
0806 12:23:45.803066 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:45.803082 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:45.803245 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:45.803927 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:45.803930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:45.803947 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:45.804067 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:45.804074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.805548 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.805552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:45.806812 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:45.806825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.808007 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.808010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:45.808117 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.808143 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 12:23:45.808747 (+ 604us) client_negotiation.cc:240] Negotiation successful
0806 12:23:45.808825 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":241,"thread_start_us":130,"threads_started":1}
I20260806 12:23:45.809021 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.802798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:39093 (local address 127.0.0.1:39808)
0806 12:23:45.803611 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:45.803624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:45.803782 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:45.804412 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:45.804415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:45.804435 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:45.804547 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:45.804553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.806053 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.806057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:45.807104 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:45.807112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.808233 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.808237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:45.808335 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.808398 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 12:23:45.808476 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 12:23:45.808547 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":2,"thread_start_us":108,"threads_started":1}
I20260806 12:23:45.809798 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:45.801953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.28.210.65:36723)
0806 12:23:45.802823 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:45.802832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:45.802863 (+ 31us) server_negotiation.cc:407] Connection header received
0806 12:23:45.802963 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:45.802969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:45.803219 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:45.803871 (+ 652us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:45.804035 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.805047 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.806696 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:45.807617 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:45.807747 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:45.807935 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:45.809493 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0806 12:23:45.809617 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":118,"threads_started":1}
I20260806 12:23:45.812737 3746 tablet_service.cc:1467] Processing CreateTablet for tablet 7b53b48f255848c1a54f5caf381dfc00 (DEFAULT_TABLE table=test-table [id=86cca8afbd6d4e718da71f22e8aa98c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:45.812737 3961 tablet_service.cc:1467] Processing CreateTablet for tablet 7b53b48f255848c1a54f5caf381dfc00 (DEFAULT_TABLE table=test-table [id=86cca8afbd6d4e718da71f22e8aa98c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:45.814116 3746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b53b48f255848c1a54f5caf381dfc00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:45.814116 3961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b53b48f255848c1a54f5caf381dfc00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:45.814025 4167 tablet_service.cc:1467] Processing CreateTablet for tablet 7b53b48f255848c1a54f5caf381dfc00 (DEFAULT_TABLE table=test-table [id=86cca8afbd6d4e718da71f22e8aa98c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:45.815325 4167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b53b48f255848c1a54f5caf381dfc00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:45.823212 4304 tablet_bootstrap.cc:492] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9: Bootstrap starting.
I20260806 12:23:45.824455 4307 tablet_bootstrap.cc:492] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f: Bootstrap starting.
I20260806 12:23:45.825131 4308 tablet_bootstrap.cc:492] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079: Bootstrap starting.
I20260806 12:23:45.827162 4307 tablet_bootstrap.cc:654] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:45.827636 4308 tablet_bootstrap.cc:654] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:45.827636 4304 tablet_bootstrap.cc:654] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:45.828312 4307 log.cc:824] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:45.829221 4308 log.cc:824] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:45.829221 4304 log.cc:824] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:45.831003 4307 tablet_bootstrap.cc:492] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f: No bootstrap required, opened a new log
I20260806 12:23:45.831248 4307 ts_tablet_manager.cc:1397] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 12:23:45.831581 4308 tablet_bootstrap.cc:492] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079: No bootstrap required, opened a new log
I20260806 12:23:45.831590 4304 tablet_bootstrap.cc:492] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9: No bootstrap required, opened a new log
I20260806 12:23:45.831811 4308 ts_tablet_manager.cc:1397] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260806 12:23:45.831817 4304 ts_tablet_manager.cc:1397] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260806 12:23:45.835737 4307 raft_consensus.cc:348] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9a8c96dc4148e4be9db3656c598c7f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36723 } } peers { permanent_uuid: "5ebcec7866684cd3bc143f4f090670c9" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 39093 } } peers { permanent_uuid: "fbab83d2837f41739171956214b03079" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37401 } }
I20260806 12:23:45.836000 4304 raft_consensus.cc:348] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9a8c96dc4148e4be9db3656c598c7f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36723 } } peers { permanent_uuid: "5ebcec7866684cd3bc143f4f090670c9" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 39093 } } peers { permanent_uuid: "fbab83d2837f41739171956214b03079" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37401 } }
I20260806 12:23:45.836000 4308 raft_consensus.cc:348] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9a8c96dc4148e4be9db3656c598c7f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36723 } } peers { permanent_uuid: "5ebcec7866684cd3bc143f4f090670c9" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 39093 } } peers { permanent_uuid: "fbab83d2837f41739171956214b03079" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37401 } }
I20260806 12:23:45.836054 4307 raft_consensus.cc:374] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:45.836323 4304 raft_consensus.cc:374] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:45.836323 4308 raft_consensus.cc:374] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:45.836349 4307 raft_consensus.cc:729] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d9a8c96dc4148e4be9db3656c598c7f, State: Initialized, Role: FOLLOWER
I20260806 12:23:45.836378 4304 raft_consensus.cc:729] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ebcec7866684cd3bc143f4f090670c9, State: Initialized, Role: FOLLOWER
I20260806 12:23:45.836378 4308 raft_consensus.cc:729] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbab83d2837f41739171956214b03079, State: Initialized, Role: FOLLOWER
I20260806 12:23:45.836884 4307 consensus_queue.cc:260] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9a8c96dc4148e4be9db3656c598c7f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36723 } } peers { permanent_uuid: "5ebcec7866684cd3bc143f4f090670c9" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 39093 } } peers { permanent_uuid: "fbab83d2837f41739171956214b03079" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37401 } }
I20260806 12:23:45.836889 4304 consensus_queue.cc:260] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9a8c96dc4148e4be9db3656c598c7f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36723 } } peers { permanent_uuid: "5ebcec7866684cd3bc143f4f090670c9" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 39093 } } peers { permanent_uuid: "fbab83d2837f41739171956214b03079" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37401 } }
I20260806 12:23:45.836889 4308 consensus_queue.cc:260] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9a8c96dc4148e4be9db3656c598c7f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36723 } } peers { permanent_uuid: "5ebcec7866684cd3bc143f4f090670c9" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 39093 } } peers { permanent_uuid: "fbab83d2837f41739171956214b03079" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37401 } }
I20260806 12:23:45.837940 4273 heartbeater.cc:503] Master 127.28.210.126:35117 was elected leader, sending a full tablet report...
I20260806 12:23:45.838076 4308 ts_tablet_manager.cc:1428] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 12:23:45.838078 4307 ts_tablet_manager.cc:1428] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 12:23:45.838371 3864 heartbeater.cc:503] Master 127.28.210.126:35117 was elected leader, sending a full tablet report...
I20260806 12:23:45.838680 4304 ts_tablet_manager.cc:1428] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260806 12:23:45.839041 4067 heartbeater.cc:503] Master 127.28.210.126:35117 was elected leader, sending a full tablet report...
W20260806 12:23:45.970676 3865 tablet.cc:2369] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:23:45.974587 4068 tablet.cc:2369] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:23:45.999325 4274 tablet.cc:2369] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:23:46.006840 4314 raft_consensus.cc:482] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:46.007176 4314 raft_consensus.cc:504] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9a8c96dc4148e4be9db3656c598c7f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36723 } } peers { permanent_uuid: "5ebcec7866684cd3bc143f4f090670c9" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 39093 } } peers { permanent_uuid: "fbab83d2837f41739171956214b03079" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37401 } }
I20260806 12:23:46.008679 4314 leader_election.cc:288] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d9a8c96dc4148e4be9db3656c598c7f (127.28.210.65:36723), fbab83d2837f41739171956214b03079 (127.28.210.66:37401)
I20260806 12:23:46.014035 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:46.009053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:36723 (local address 127.28.210.67:52359)
0806 12:23:46.009209 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:46.009228 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:46.009361 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:46.009702 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:46.009706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:46.009734 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:46.009903 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:46.009912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:46.011008 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:46.011011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:46.012403 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:46.012418 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:46.013429 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:46.013435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:46.013571 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:46.013627 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 12:23:46.013725 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 12:23:46.013888 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":30}
I20260806 12:23:46.014101 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:46.009135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:52359 (local address 127.28.210.65:36723)
0806 12:23:46.009289 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:46.009294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:46.009316 (+ 22us) server_negotiation.cc:407] Connection header received
0806 12:23:46.009494 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:46.009498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:46.009558 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:46.009645 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:46.010090 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:46.010859 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:46.012567 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:46.013259 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:46.013412 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:46.013480 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:46.013849 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 12:23:46.013944 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 12:23:46.014101 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:46.009171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:39499 (local address 127.28.210.66:37401)
0806 12:23:46.009300 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:46.009305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:46.010063 (+ 758us) server_negotiation.cc:407] Connection header received
0806 12:23:46.010125 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:46.010128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:46.010179 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:46.010288 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:46.010578 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:46.011198 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:46.012747 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:46.013374 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:46.013492 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:46.013536 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:46.013820 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 12:23:46.013944 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 12:23:46.014048 4320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:46.009071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:37401 (local address 127.28.210.67:39499)
0806 12:23:46.009944 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:46.009958 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:46.010060 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:46.010300 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:46.010302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:46.010313 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:46.010439 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:46.010443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:46.011366 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:46.011371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:46.012620 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:46.012629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:46.013485 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:46.013488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:46.013576 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:46.013653 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 12:23:46.013725 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 12:23:46.013846 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":503,"thread_start_us":190,"threads_started":1}
I20260806 12:23:46.015179 3784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b53b48f255848c1a54f5caf381dfc00" candidate_uuid: "5ebcec7866684cd3bc143f4f090670c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d9a8c96dc4148e4be9db3656c598c7f" is_pre_election: true
I20260806 12:23:46.015319 3990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b53b48f255848c1a54f5caf381dfc00" candidate_uuid: "5ebcec7866684cd3bc143f4f090670c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbab83d2837f41739171956214b03079" is_pre_election: true
I20260806 12:23:46.015657 3784 raft_consensus.cc:2454] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ebcec7866684cd3bc143f4f090670c9 in term 0.
I20260806 12:23:46.015733 3990 raft_consensus.cc:2454] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ebcec7866684cd3bc143f4f090670c9 in term 0.
I20260806 12:23:46.016387 4097 leader_election.cc:302] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d9a8c96dc4148e4be9db3656c598c7f, 5ebcec7866684cd3bc143f4f090670c9; no voters:
I20260806 12:23:46.016865 4314 raft_consensus.cc:2781] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:23:46.016966 4314 raft_consensus.cc:482] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:46.016997 4314 raft_consensus.cc:3037] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:46.018175 4314 raft_consensus.cc:504] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9a8c96dc4148e4be9db3656c598c7f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36723 } } peers { permanent_uuid: "5ebcec7866684cd3bc143f4f090670c9" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 39093 } } peers { permanent_uuid: "fbab83d2837f41739171956214b03079" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37401 } }
I20260806 12:23:46.018579 4314 leader_election.cc:288] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [CANDIDATE]: Term 1 election: Requested vote from peers 4d9a8c96dc4148e4be9db3656c598c7f (127.28.210.65:36723), fbab83d2837f41739171956214b03079 (127.28.210.66:37401)
I20260806 12:23:46.019173 3990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b53b48f255848c1a54f5caf381dfc00" candidate_uuid: "5ebcec7866684cd3bc143f4f090670c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbab83d2837f41739171956214b03079"
I20260806 12:23:46.019325 3990 raft_consensus.cc:3037] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:46.020694 3784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b53b48f255848c1a54f5caf381dfc00" candidate_uuid: "5ebcec7866684cd3bc143f4f090670c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d9a8c96dc4148e4be9db3656c598c7f"
I20260806 12:23:46.020980 3990 raft_consensus.cc:2454] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ebcec7866684cd3bc143f4f090670c9 in term 1.
I20260806 12:23:46.020982 3784 raft_consensus.cc:3037] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:46.021533 4101 leader_election.cc:302] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ebcec7866684cd3bc143f4f090670c9, fbab83d2837f41739171956214b03079; no voters:
I20260806 12:23:46.021837 4314 raft_consensus.cc:2781] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:46.022425 4314 raft_consensus.cc:686] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5ebcec7866684cd3bc143f4f090670c9, State: Running, Role: LEADER
I20260806 12:23:46.023049 4314 consensus_queue.cc:237] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9a8c96dc4148e4be9db3656c598c7f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36723 } } peers { permanent_uuid: "5ebcec7866684cd3bc143f4f090670c9" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 39093 } } peers { permanent_uuid: "fbab83d2837f41739171956214b03079" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37401 } }
I20260806 12:23:46.023718 3784 raft_consensus.cc:2454] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ebcec7866684cd3bc143f4f090670c9 in term 1.
I20260806 12:23:46.027772 3544 catalog_manager.cc:5495] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ebcec7866684cd3bc143f4f090670c9 (127.28.210.67). New cstate: current_term: 1 leader_uuid: "5ebcec7866684cd3bc143f4f090670c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9a8c96dc4148e4be9db3656c598c7f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ebcec7866684cd3bc143f4f090670c9" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 39093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbab83d2837f41739171956214b03079" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260806 12:23:46.051158 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:46.047186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.28.210.67:39093)
0806 12:23:46.047345 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:46.047349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:46.047364 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:23:46.047451 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:46.047454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:46.047515 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:46.047619 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:46.047863 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:46.048406 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:46.049376 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:46.049611 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:46.049704 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:46.050114 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:46.050626 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:46.050751 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:46.050930 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 12:23:46.051001 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 12:23:46.058585 3784 raft_consensus.cc:1261] T 7b53b48f255848c1a54f5caf381dfc00 P 4d9a8c96dc4148e4be9db3656c598c7f [term 1 FOLLOWER]: Refusing update from remote peer 5ebcec7866684cd3bc143f4f090670c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:46.058585 3990 raft_consensus.cc:1261] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079 [term 1 FOLLOWER]: Refusing update from remote peer 5ebcec7866684cd3bc143f4f090670c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:46.059528 4314 consensus_queue.cc:1035] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d9a8c96dc4148e4be9db3656c598c7f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:46.059804 4321 consensus_queue.cc:1035] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbab83d2837f41739171956214b03079" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:46.069638 4327 mvcc.cc:205] Tried to move back new op lower bound from 7315533930725392384 to 7315533930594811904. Current Snapshot: MvccSnapshot[applied={T|T < 7315533930725392384}]
I20260806 12:23:46.071765 4326 mvcc.cc:205] Tried to move back new op lower bound from 7315533930725392384 to 7315533930594811904. Current Snapshot: MvccSnapshot[applied={T|T < 7315533930725392384}]
I20260806 12:23:46.079257 3542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:46.079478 3542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:46.082749 3542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b643d873a49e44bea949099c2a3b7bac: Sending DeleteTablet for 3 replicas of tablet 7b53b48f255848c1a54f5caf381dfc00
I20260806 12:23:46.083961 3961 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b53b48f255848c1a54f5caf381dfc00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:46 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:43320
I20260806 12:23:46.084066 4167 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b53b48f255848c1a54f5caf381dfc00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:46 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:39808
I20260806 12:23:46.084134 3746 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b53b48f255848c1a54f5caf381dfc00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:46 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:50074
I20260806 12:23:46.084715 4339 tablet_replica.cc:330] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9: stopping tablet replica
I20260806 12:23:46.084973 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 3648
I20260806 12:23:46.085132 4339 raft_consensus.cc:2229] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:23:46.085937 4339 raft_consensus.cc:2258] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:46.109290 4338 tablet_replica.cc:330] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079: stopping tablet replica
I20260806 12:23:46.110692 4338 raft_consensus.cc:2229] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:46.128404 4338 raft_consensus.cc:2258] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:46.130301 4339 ts_tablet_manager.cc:1905] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:46.130614 4338 ts_tablet_manager.cc:1905] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:46.134105 4339 ts_tablet_manager.cc:1918] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:23:46.134260 4339 log.cc:1196] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/7b53b48f255848c1a54f5caf381dfc00
I20260806 12:23:46.134490 4338 ts_tablet_manager.cc:1918] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:23:46.134622 4339 ts_tablet_manager.cc:1939] T 7b53b48f255848c1a54f5caf381dfc00 P 5ebcec7866684cd3bc143f4f090670c9: Deleting consensus metadata
I20260806 12:23:46.134792 4338 log.cc:1196] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/7b53b48f255848c1a54f5caf381dfc00
I20260806 12:23:46.135705 4338 ts_tablet_manager.cc:1939] T 7b53b48f255848c1a54f5caf381dfc00 P fbab83d2837f41739171956214b03079: Deleting consensus metadata
W20260806 12:23:46.137292 3518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:36723 (error 108)
I20260806 12:23:46.138249 3521 catalog_manager.cc:4841] TS fbab83d2837f41739171956214b03079 (127.28.210.66:37401): tablet 7b53b48f255848c1a54f5caf381dfc00 (table test-table [id=86cca8afbd6d4e718da71f22e8aa98c8]) successfully deleted
I20260806 12:23:46.138576 3518 catalog_manager.cc:4841] TS 5ebcec7866684cd3bc143f4f090670c9 (127.28.210.67:39093): tablet 7b53b48f255848c1a54f5caf381dfc00 (table test-table [id=86cca8afbd6d4e718da71f22e8aa98c8]) successfully deleted
I20260806 12:23:46.139219 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 3869
I20260806 12:23:46.154340 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:46.153780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:36723 (local address 127.0.0.1:50080)
0806 12:23:46.154100 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:46.154199 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
W20260806 12:23:46.155772 3518 catalog_manager.cc:4584] TS 4d9a8c96dc4148e4be9db3656c598c7f (127.28.210.65:36723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b53b48f255848c1a54f5caf381dfc00: Network error: Client connection negotiation failed: client connection to 127.28.210.65:36723: connect: Connection refused (error 111)
I20260806 12:23:46.165723 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 4077
I20260806 12:23:46.183588 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 3492
2026-08-06T12:23:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2900 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T12:23:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:46Z Disabled control of system clock
I20260806 12:23:46.231468 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:44903
--webserver_interface=127.28.210.126
--webserver_port=0
--builtin_ntp_servers=127.28.210.84:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:44903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 12:23:46.362573 4348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:46.362947 4348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:46.362993 4348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:46.367568 4348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:46.367703 4348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:46.367718 4348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:46.367735 4348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:46.367749 4348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:46.373044 4348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:44903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:44903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:46.374289 4348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:46.376056 4348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:46.386706 4361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:46.386706 4359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:46.386780 4348 server_base.cc:1034] running on GCE node
W20260806 12:23:46.386799 4358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:46.387670 4348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:46.389065 4348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:46.390269 4348 hybrid_clock.cc:648] HybridClock initialized: now 1786019026390250 us; error 40 us; skew 500 ppm
I20260806 12:23:46.392630 4348 webserver.cc:460] Webserver started at http://127.28.210.126:35479/ using document root <none> and password file <none>
I20260806 12:23:46.393424 4348 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:46.393532 4348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:46.393781 4348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:46.396377 4348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "f7cc2f3ba99f4fdb9a81a3abbf00c088"
format_stamp: "Formatted at 2026-08-06 12:23:46 on dist-test-slave-lgr5"
server_key: "b5b357af1bf2816dedd13cb266005179"
server_key_iv: "bf43e8627536cb717e23c6cac5d4703c"
server_key_version: "encryptionkey@0"
I20260806 12:23:46.397048 4348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "f7cc2f3ba99f4fdb9a81a3abbf00c088"
format_stamp: "Formatted at 2026-08-06 12:23:46 on dist-test-slave-lgr5"
server_key: "b5b357af1bf2816dedd13cb266005179"
server_key_iv: "bf43e8627536cb717e23c6cac5d4703c"
server_key_version: "encryptionkey@0"
I20260806 12:23:46.401651 4348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:23:46.404702 4370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:46.406081 4348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:46.406263 4348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "f7cc2f3ba99f4fdb9a81a3abbf00c088"
format_stamp: "Formatted at 2026-08-06 12:23:46 on dist-test-slave-lgr5"
server_key: "b5b357af1bf2816dedd13cb266005179"
server_key_iv: "bf43e8627536cb717e23c6cac5d4703c"
server_key_version: "encryptionkey@0"
I20260806 12:23:46.406450 4348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:46.411936 4348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:46.412851 4348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:46.413084 4348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:46.421739 4348 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:44903
I20260806 12:23:46.421758 4460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:44903 every 8 connection(s)
I20260806 12:23:46.422945 4348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:46.426863 4466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:46.429737 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 4348
I20260806 12:23:46.429872 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:46.430176 29513 external_mini_cluster.cc:1428] Setting key 9f997d8531d8ab47c7fb16984c2a7b53
I20260806 12:23:46.433976 4466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088: Bootstrap starting.
I20260806 12:23:46.436693 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:46.431968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43144 (local address 127.28.210.126:44903)
0806 12:23:46.432746 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:46.432762 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:46.432802 (+ 40us) server_negotiation.cc:407] Connection header received
0806 12:23:46.433482 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:46.433501 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:46.433887 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:46.434245 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:46.435234 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:46.435239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:46.435253 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:46.435265 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:46.435329 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:46.435346 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:46.435496 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:46.435563 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 12:23:46.435849 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":167,"threads_started":1}
I20260806 12:23:46.436795 4466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:46.437621 4466 log.cc:824] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:46.439788 4466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088: No bootstrap required, opened a new log
I20260806 12:23:46.443600 4466 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cc2f3ba99f4fdb9a81a3abbf00c088" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44903 } }
I20260806 12:23:46.443933 4466 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:46.444007 4466 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cc2f3ba99f4fdb9a81a3abbf00c088, State: Initialized, Role: FOLLOWER
I20260806 12:23:46.444518 4466 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cc2f3ba99f4fdb9a81a3abbf00c088" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44903 } }
I20260806 12:23:46.444669 4466 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:46.444738 4466 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:46.444845 4466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:46.445969 4466 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cc2f3ba99f4fdb9a81a3abbf00c088" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44903 } }
I20260806 12:23:46.446357 4466 leader_election.cc:302] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7cc2f3ba99f4fdb9a81a3abbf00c088; no voters:
I20260806 12:23:46.446702 4466 leader_election.cc:288] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:46.446836 4475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:46.447085 4475 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [term 1 LEADER]: Becoming Leader. State: Replica: f7cc2f3ba99f4fdb9a81a3abbf00c088, State: Running, Role: LEADER
I20260806 12:23:46.447427 4475 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cc2f3ba99f4fdb9a81a3abbf00c088" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44903 } }
I20260806 12:23:46.448048 4466 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:46.449173 4478 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7cc2f3ba99f4fdb9a81a3abbf00c088. Latest consensus state: current_term: 1 leader_uuid: "f7cc2f3ba99f4fdb9a81a3abbf00c088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cc2f3ba99f4fdb9a81a3abbf00c088" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44903 } } }
I20260806 12:23:46.449309 4478 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:46.449602 4476 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7cc2f3ba99f4fdb9a81a3abbf00c088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cc2f3ba99f4fdb9a81a3abbf00c088" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44903 } } }
I20260806 12:23:46.449695 4476 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:46.449792 4492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:46.453152 4492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:46.459560 4492 catalog_manager.cc:1347] Generated new cluster ID: ea7813cfe47f4f658fca3ddfa852d305
I20260806 12:23:46.459654 4492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:46.474210 4492 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:46.474656 4492 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:46.485371 4492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088: Generated new TSK 0
I20260806 12:23:46.486234 4492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:23:46.493034 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:44903
--builtin_ntp_servers=127.28.210.84:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 12:23:46.634052 4511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:46.634392 4511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:46.634438 4511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:46.638967 4511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:46.639058 4511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:46.639123 4511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:23:46.644454 4511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:44903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:46.645687 4511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:46.647200 4511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:23:46.658555 4511 server_base.cc:1034] running on GCE node
W20260806 12:23:46.658475 4520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:46.658433 4527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:46.658499 4521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:46.659232 4511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:46.659956 4511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:46.661161 4511 hybrid_clock.cc:648] HybridClock initialized: now 1786019026661125 us; error 48 us; skew 500 ppm
I20260806 12:23:46.663595 4511 webserver.cc:460] Webserver started at http://127.28.210.65:42903/ using document root <none> and password file <none>
I20260806 12:23:46.664358 4511 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:46.664449 4511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:46.664698 4511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:46.667193 4511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "1ba8f0498aca43808ccd934c640c247a"
format_stamp: "Formatted at 2026-08-06 12:23:46 on dist-test-slave-lgr5"
server_key: "d74d839c1979f5fbf52e92e5aded4923"
server_key_iv: "12dba3d7e0d687bd73a76708125ccdb6"
server_key_version: "encryptionkey@0"
I20260806 12:23:46.667912 4511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "1ba8f0498aca43808ccd934c640c247a"
format_stamp: "Formatted at 2026-08-06 12:23:46 on dist-test-slave-lgr5"
server_key: "d74d839c1979f5fbf52e92e5aded4923"
server_key_iv: "12dba3d7e0d687bd73a76708125ccdb6"
server_key_version: "encryptionkey@0"
I20260806 12:23:46.672669 4511 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 12:23:46.676069 4537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:46.677977 4511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:23:46.678182 4511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "1ba8f0498aca43808ccd934c640c247a"
format_stamp: "Formatted at 2026-08-06 12:23:46 on dist-test-slave-lgr5"
server_key: "d74d839c1979f5fbf52e92e5aded4923"
server_key_iv: "12dba3d7e0d687bd73a76708125ccdb6"
server_key_version: "encryptionkey@0"
I20260806 12:23:46.678402 4511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:46.684249 4511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:46.685129 4511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:46.685367 4511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:46.686084 4511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:46.687287 4511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:46.687351 4511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:46.687402 4511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:46.687419 4511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:46.697806 4511 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:39239
I20260806 12:23:46.697830 4714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:39239 every 8 connection(s)
I20260806 12:23:46.699086 4511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:23:46.703509 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 4511
I20260806 12:23:46.703676 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:23:46.704007 29513 external_mini_cluster.cc:1428] Setting key fd67a9b63353dfd1df04b8cf87c76309
I20260806 12:23:46.706815 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:44903
--builtin_ntp_servers=127.28.210.84:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 12:23:46.707650 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:46.701566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:59651 (local address 127.28.210.126:44903)
0806 12:23:46.701758 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:46.701764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:46.703334 (+ 1570us) server_negotiation.cc:407] Connection header received
0806 12:23:46.705328 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:46.705332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:46.705379 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:46.705480 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:46.707151 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:46.707155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:46.707158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:46.707178 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:46.707202 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:46.707213 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:46.707310 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:46.707476 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 12:23:46.707523 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 12:23:46.709229 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:46.702055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:44903 (local address 127.28.210.65:59651)
0806 12:23:46.703040 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:46.703094 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:46.705036 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:46.705826 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:46.705838 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:46.705985 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:46.706680 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:23:46.706682 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:46.706755 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:23:46.706880 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:46.707312 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:46.707323 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:46.707329 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 12:23:46.707551 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 12:23:46.707914 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":236,"threads_started":1}
I20260806 12:23:46.711093 4715 heartbeater.cc:347] Connected to a master server at 127.28.210.126:44903
I20260806 12:23:46.711424 4715 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:46.712049 4715 heartbeater.cc:511] Master 127.28.210.126:44903 requested a full tablet report, sending...
I20260806 12:23:46.714056 4405 ts_manager.cc:194] Registered new tserver with Master: 1ba8f0498aca43808ccd934c640c247a (127.28.210.65:39239)
W20260806 12:23:46.841135 4719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:46.841542 4719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:46.841609 4719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:46.846329 4719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:46.846410 4719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:46.846494 4719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:23:46.852277 4719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:44903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:46.853672 4719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:46.855394 4719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:46.866367 4732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:46.866372 4729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:46.866376 4730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:46.866633 4719 server_base.cc:1034] running on GCE node
I20260806 12:23:46.867162 4719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:46.867895 4719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:46.869077 4719 hybrid_clock.cc:648] HybridClock initialized: now 1786019026869049 us; error 46 us; skew 500 ppm
I20260806 12:23:46.871220 4719 webserver.cc:460] Webserver started at http://127.28.210.66:37409/ using document root <none> and password file <none>
I20260806 12:23:46.871866 4719 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:46.871944 4719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:46.872207 4719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:46.874576 4719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "5f8248660b54416380a470eed2acda89"
format_stamp: "Formatted at 2026-08-06 12:23:46 on dist-test-slave-lgr5"
server_key: "16451e9cbabcfc590daa902396817c2d"
server_key_iv: "8390e126d5a82dd90cd37b8750629106"
server_key_version: "encryptionkey@0"
I20260806 12:23:46.875233 4719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "5f8248660b54416380a470eed2acda89"
format_stamp: "Formatted at 2026-08-06 12:23:46 on dist-test-slave-lgr5"
server_key: "16451e9cbabcfc590daa902396817c2d"
server_key_iv: "8390e126d5a82dd90cd37b8750629106"
server_key_version: "encryptionkey@0"
I20260806 12:23:46.879647 4719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 12:23:46.883460 4739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:46.885263 4719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:23:46.885460 4719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "5f8248660b54416380a470eed2acda89"
format_stamp: "Formatted at 2026-08-06 12:23:46 on dist-test-slave-lgr5"
server_key: "16451e9cbabcfc590daa902396817c2d"
server_key_iv: "8390e126d5a82dd90cd37b8750629106"
server_key_version: "encryptionkey@0"
I20260806 12:23:46.885619 4719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:46.891292 4719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:46.892168 4719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:46.892385 4719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:46.893153 4719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:46.894347 4719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:46.894413 4719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:46.894490 4719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:46.894518 4719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:46.904395 4719 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:41339
I20260806 12:23:46.904448 4926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:41339 every 8 connection(s)
I20260806 12:23:46.905514 4719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:23:46.907272 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 4719
I20260806 12:23:46.907415 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:23:46.907780 29513 external_mini_cluster.cc:1428] Setting key 3c6f34b69096d6732780ba09bcab5607
I20260806 12:23:46.910419 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:44903
--builtin_ntp_servers=127.28.210.84:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 12:23:46.913902 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:46.907848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:59219 (local address 127.28.210.126:44903)
0806 12:23:46.908036 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:46.908041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:46.909763 (+ 1722us) server_negotiation.cc:407] Connection header received
0806 12:23:46.911524 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:46.911529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:46.911588 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:46.911699 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:46.913258 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:46.913262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:46.913265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:46.913282 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:46.913309 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:46.913319 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:46.913421 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:46.913674 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 12:23:46.913738 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 12:23:46.914958 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:46.908479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:44903 (local address 127.28.210.66:59219)
0806 12:23:46.909560 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:46.909637 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:46.911199 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:46.911915 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:46.911934 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:46.912058 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:46.912821 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:23:46.912823 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:46.912895 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:23:46.913033 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:46.913415 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:46.913423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:46.913431 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 12:23:46.913658 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 12:23:46.914017 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":173,"threads_started":1}
I20260806 12:23:46.916780 4927 heartbeater.cc:347] Connected to a master server at 127.28.210.126:44903
I20260806 12:23:46.917107 4927 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:46.917644 4927 heartbeater.cc:511] Master 127.28.210.126:44903 requested a full tablet report, sending...
I20260806 12:23:46.919101 4405 ts_manager.cc:194] Registered new tserver with Master: 5f8248660b54416380a470eed2acda89 (127.28.210.66:41339)
W20260806 12:23:47.050217 4932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:47.050563 4932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:47.050611 4932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:47.055184 4932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:47.055279 4932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:47.055343 4932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:23:47.060667 4932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:44903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:47.062075 4932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:47.063815 4932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:47.076925 4943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:47.076929 4942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:47.076928 4947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:47.077046 4932 server_base.cc:1034] running on GCE node
I20260806 12:23:47.078111 4932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:47.079039 4932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:47.080201 4932 hybrid_clock.cc:648] HybridClock initialized: now 1786019027080165 us; error 57 us; skew 500 ppm
I20260806 12:23:47.082643 4932 webserver.cc:460] Webserver started at http://127.28.210.67:40641/ using document root <none> and password file <none>
I20260806 12:23:47.083395 4932 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:47.083494 4932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:47.083752 4932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:47.086242 4932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "0b993543874840ec82a7d712c89e69a3"
format_stamp: "Formatted at 2026-08-06 12:23:47 on dist-test-slave-lgr5"
server_key: "9403dbcdbc042678d67fe56251a81710"
server_key_iv: "c175db77c792724aef9d10184e91eb39"
server_key_version: "encryptionkey@0"
I20260806 12:23:47.086935 4932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "0b993543874840ec82a7d712c89e69a3"
format_stamp: "Formatted at 2026-08-06 12:23:47 on dist-test-slave-lgr5"
server_key: "9403dbcdbc042678d67fe56251a81710"
server_key_iv: "c175db77c792724aef9d10184e91eb39"
server_key_version: "encryptionkey@0"
I20260806 12:23:47.091979 4932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 12:23:47.096266 4954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:47.098265 4932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260806 12:23:47.098471 4932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "0b993543874840ec82a7d712c89e69a3"
format_stamp: "Formatted at 2026-08-06 12:23:47 on dist-test-slave-lgr5"
server_key: "9403dbcdbc042678d67fe56251a81710"
server_key_iv: "c175db77c792724aef9d10184e91eb39"
server_key_version: "encryptionkey@0"
I20260806 12:23:47.098635 4932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:47.104852 4932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:47.105762 4932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:47.106000 4932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:47.106806 4932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:47.108119 4932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:47.108186 4932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:47.108253 4932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:47.108278 4932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:47.122428 4932 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:44195
I20260806 12:23:47.122538 5129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:44195 every 8 connection(s)
I20260806 12:23:47.123978 4932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:23:47.132304 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 4932
I20260806 12:23:47.132447 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:23:47.132591 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.126787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:47275 (local address 127.28.210.126:44903)
0806 12:23:47.127150 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:47.127155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:47.128364 (+ 1209us) server_negotiation.cc:407] Connection header received
0806 12:23:47.129891 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:47.129894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:47.129951 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:47.130048 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:47.131775 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:47.131780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:47.131782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:47.131820 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:47.131849 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:47.131859 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:47.132068 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:47.132336 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 12:23:47.132407 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 12:23:47.132777 29513 external_mini_cluster.cc:1428] Setting key be29f1e7962e0c52fc55cf487b823d3a
I20260806 12:23:47.134078 5133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.127123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:44903 (local address 127.28.210.67:47275)
0806 12:23:47.128156 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:47.128217 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:47.129594 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:47.130352 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:47.130362 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:47.130479 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:47.131228 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:23:47.131230 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:47.131295 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:23:47.131392 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:47.132129 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:47.132145 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:47.132155 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 12:23:47.132399 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 12:23:47.132733 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":176,"threads_started":1}
I20260806 12:23:47.135634 5130 heartbeater.cc:347] Connected to a master server at 127.28.210.126:44903
I20260806 12:23:47.135928 5130 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:47.136489 5130 heartbeater.cc:511] Master 127.28.210.126:44903 requested a full tablet report, sending...
I20260806 12:23:47.137789 4405 ts_manager.cc:194] Registered new tserver with Master: 0b993543874840ec82a7d712c89e69a3 (127.28.210.67:44195)
I20260806 12:23:47.148664 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:23:47.153484 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.151308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43158 (local address 127.28.210.126:44903)
0806 12:23:47.151599 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:47.151604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:47.151725 (+ 121us) server_negotiation.cc:407] Connection header received
0806 12:23:47.151929 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:47.151933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:47.151981 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:47.152107 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:47.153054 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:47.153057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:47.153059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:47.153072 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:47.153094 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:47.153102 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:47.153186 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:47.153275 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 12:23:47.153323 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 12:23:47.157308 4405 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:23:47.160854 4405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:47.175258 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.171102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:41339 (local address 127.0.0.1:59304)
0806 12:23:47.172065 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:47.172156 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:47.172270 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:47.173076 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:47.173083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:47.173167 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:47.173907 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:23:47.173909 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:47.173935 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:23:47.174016 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:47.174812 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:47.174815 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:47.174823 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 12:23:47.174973 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 12:23:47.175078 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":136,"threads_started":1}
I20260806 12:23:47.175303 5159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.171836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54262 (local address 127.28.210.67:44195)
0806 12:23:47.172546 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:47.172559 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:47.173005 (+ 446us) server_negotiation.cc:407] Connection header received
0806 12:23:47.173145 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:47.173151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:47.173335 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:47.173456 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:47.174512 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:47.174519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:47.174535 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:47.174548 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:47.174600 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:47.174637 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:47.174769 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:47.175050 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 12:23:47.175140 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":127,"threads_started":1}
I20260806 12:23:47.175623 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.171699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:39239 (local address 127.0.0.1:45172)
0806 12:23:47.173167 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:47.173182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:47.173330 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:47.174062 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:47.174065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:47.174080 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:47.174835 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:23:47.174837 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:47.174848 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:23:47.174873 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:47.175315 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:47.175318 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:47.175321 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 12:23:47.175409 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 12:23:47.175491 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"thread_start_us":102,"threads_started":1}
I20260806 12:23:47.176079 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.171308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.28.210.66:41339)
0806 12:23:47.172364 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:47.172382 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:47.172413 (+ 31us) server_negotiation.cc:407] Connection header received
0806 12:23:47.172475 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:47.172481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:47.172884 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:47.173058 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:47.174605 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:47.174610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:47.174620 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:47.174634 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:47.174673 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:47.174707 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:47.174809 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:47.175791 (+ 982us) server_negotiation.cc:299] Negotiation successful
0806 12:23:47.175883 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939,"thread_start_us":115,"threads_started":1}
I20260806 12:23:47.176139 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.172187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.28.210.65:39239)
0806 12:23:47.172906 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:47.172925 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:47.173451 (+ 526us) server_negotiation.cc:407] Connection header received
0806 12:23:47.173524 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:47.173529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:47.173712 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:47.173858 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:47.175035 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:47.175041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:47.175052 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:47.175065 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:47.175114 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:47.175155 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:47.175261 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:47.175865 (+ 604us) server_negotiation.cc:299] Negotiation successful
0806 12:23:47.175955 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":124,"threads_started":1}
I20260806 12:23:47.176502 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.171386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:44195 (local address 127.0.0.1:54262)
0806 12:23:47.172869 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:47.172895 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:47.173009 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:47.173849 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:47.173852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:47.173867 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:47.174341 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:23:47.174343 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:47.174352 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:23:47.174382 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:47.174800 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:47.174814 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:47.174823 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 12:23:47.176108 (+ 1285us) client_negotiation.cc:240] Negotiation successful
0806 12:23:47.176396 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":137,"threads_started":1}
I20260806 12:23:47.179454 4606 tablet_service.cc:1467] Processing CreateTablet for tablet eeb6dcb2d0844e22ac823d5ad7fc555b (DEFAULT_TABLE table=test-table [id=eb7d88ed04f44695b5127e58b7977f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:47.179533 4811 tablet_service.cc:1467] Processing CreateTablet for tablet eeb6dcb2d0844e22ac823d5ad7fc555b (DEFAULT_TABLE table=test-table [id=eb7d88ed04f44695b5127e58b7977f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:47.179533 5026 tablet_service.cc:1467] Processing CreateTablet for tablet eeb6dcb2d0844e22ac823d5ad7fc555b (DEFAULT_TABLE table=test-table [id=eb7d88ed04f44695b5127e58b7977f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:47.180860 4606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb6dcb2d0844e22ac823d5ad7fc555b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:47.181001 4811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb6dcb2d0844e22ac823d5ad7fc555b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:47.181604 5026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb6dcb2d0844e22ac823d5ad7fc555b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:47.189359 5162 tablet_bootstrap.cc:492] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a: Bootstrap starting.
I20260806 12:23:47.191174 5168 tablet_bootstrap.cc:492] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3: Bootstrap starting.
I20260806 12:23:47.191164 5163 tablet_bootstrap.cc:492] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89: Bootstrap starting.
I20260806 12:23:47.193186 5162 tablet_bootstrap.cc:654] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:47.193670 5163 tablet_bootstrap.cc:654] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:47.193670 5168 tablet_bootstrap.cc:654] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:47.194115 5162 log.cc:824] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:47.194593 5163 log.cc:824] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:47.194592 5168 log.cc:824] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:47.196616 5163 tablet_bootstrap.cc:492] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89: No bootstrap required, opened a new log
I20260806 12:23:47.196831 5163 ts_tablet_manager.cc:1397] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 12:23:47.197176 5162 tablet_bootstrap.cc:492] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a: No bootstrap required, opened a new log
I20260806 12:23:47.197434 5162 ts_tablet_manager.cc:1397] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260806 12:23:47.199405 5168 tablet_bootstrap.cc:492] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3: No bootstrap required, opened a new log
I20260806 12:23:47.199728 5168 ts_tablet_manager.cc:1397] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260806 12:23:47.201784 5162 raft_consensus.cc:348] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8248660b54416380a470eed2acda89" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41339 } } peers { permanent_uuid: "0b993543874840ec82a7d712c89e69a3" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 44195 } } peers { permanent_uuid: "1ba8f0498aca43808ccd934c640c247a" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 39239 } }
I20260806 12:23:47.202112 5162 raft_consensus.cc:374] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:47.202188 5162 raft_consensus.cc:729] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba8f0498aca43808ccd934c640c247a, State: Initialized, Role: FOLLOWER
I20260806 12:23:47.202203 5163 raft_consensus.cc:348] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8248660b54416380a470eed2acda89" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41339 } } peers { permanent_uuid: "0b993543874840ec82a7d712c89e69a3" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 44195 } } peers { permanent_uuid: "1ba8f0498aca43808ccd934c640c247a" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 39239 } }
I20260806 12:23:47.202549 5163 raft_consensus.cc:374] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:47.202618 5163 raft_consensus.cc:729] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8248660b54416380a470eed2acda89, State: Initialized, Role: FOLLOWER
I20260806 12:23:47.202816 5162 consensus_queue.cc:260] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8248660b54416380a470eed2acda89" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41339 } } peers { permanent_uuid: "0b993543874840ec82a7d712c89e69a3" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 44195 } } peers { permanent_uuid: "1ba8f0498aca43808ccd934c640c247a" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 39239 } }
I20260806 12:23:47.203158 5163 consensus_queue.cc:260] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8248660b54416380a470eed2acda89" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41339 } } peers { permanent_uuid: "0b993543874840ec82a7d712c89e69a3" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 44195 } } peers { permanent_uuid: "1ba8f0498aca43808ccd934c640c247a" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 39239 } }
I20260806 12:23:47.204227 5162 ts_tablet_manager.cc:1428] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260806 12:23:47.204433 5163 ts_tablet_manager.cc:1428] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 12:23:47.204671 4715 heartbeater.cc:503] Master 127.28.210.126:44903 was elected leader, sending a full tablet report...
I20260806 12:23:47.204952 4927 heartbeater.cc:503] Master 127.28.210.126:44903 was elected leader, sending a full tablet report...
I20260806 12:23:47.206849 5168 raft_consensus.cc:348] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8248660b54416380a470eed2acda89" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41339 } } peers { permanent_uuid: "0b993543874840ec82a7d712c89e69a3" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 44195 } } peers { permanent_uuid: "1ba8f0498aca43808ccd934c640c247a" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 39239 } }
I20260806 12:23:47.207226 5168 raft_consensus.cc:374] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:47.207317 5168 raft_consensus.cc:729] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b993543874840ec82a7d712c89e69a3, State: Initialized, Role: FOLLOWER
I20260806 12:23:47.207885 5168 consensus_queue.cc:260] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8248660b54416380a470eed2acda89" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41339 } } peers { permanent_uuid: "0b993543874840ec82a7d712c89e69a3" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 44195 } } peers { permanent_uuid: "1ba8f0498aca43808ccd934c640c247a" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 39239 } }
I20260806 12:23:47.208973 5130 heartbeater.cc:503] Master 127.28.210.126:44903 was elected leader, sending a full tablet report...
I20260806 12:23:47.209383 5168 ts_tablet_manager.cc:1428] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
W20260806 12:23:47.375505 5132 tablet.cc:2369] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:23:47.405812 5176 raft_consensus.cc:482] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:47.406081 5176 raft_consensus.cc:504] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8248660b54416380a470eed2acda89" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41339 } } peers { permanent_uuid: "0b993543874840ec82a7d712c89e69a3" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 44195 } } peers { permanent_uuid: "1ba8f0498aca43808ccd934c640c247a" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 39239 } }
W20260806 12:23:47.406880 4928 tablet.cc:2369] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:23:47.407429 5176 leader_election.cc:288] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8248660b54416380a470eed2acda89 (127.28.210.66:41339), 1ba8f0498aca43808ccd934c640c247a (127.28.210.65:39239)
I20260806 12:23:47.409304 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.407805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:35445 (local address 127.28.210.66:41339)
0806 12:23:47.407931 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:47.407936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:47.407954 (+ 18us) server_negotiation.cc:407] Connection header received
0806 12:23:47.408129 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:47.408132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:47.408177 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:47.408255 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:47.408932 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:47.408937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:47.408940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:47.408954 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:47.408978 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:47.408989 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:47.409083 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:47.409124 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 12:23:47.409170 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 12:23:47.409304 5133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.407750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:41339 (local address 127.28.210.67:35445)
0806 12:23:47.407886 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:47.407900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:47.407997 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:47.408298 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:47.408301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:47.408316 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:47.408770 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:23:47.408772 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:47.408785 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:23:47.408816 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:47.409063 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:47.409065 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:47.409068 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 12:23:47.409130 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 12:23:47.409169 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 12:23:47.409336 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.407900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:52897 (local address 127.28.210.65:39239)
0806 12:23:47.408032 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:47.408036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:47.408315 (+ 279us) server_negotiation.cc:407] Connection header received
0806 12:23:47.408378 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:47.408380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:47.408412 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:47.408474 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:47.408999 (+ 525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:47.409003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:47.409005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:47.409018 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:47.409037 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:47.409046 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:47.409124 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:47.409193 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 12:23:47.409230 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 12:23:47.409317 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.407766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:39239 (local address 127.28.210.67:52897)
0806 12:23:47.408211 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:47.408222 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:47.408310 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:47.408513 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:47.408516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:47.408528 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:47.408854 (+ 326us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:23:47.408856 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:47.408864 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:23:47.408885 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:47.409116 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:47.409119 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:47.409122 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 12:23:47.409186 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 12:23:47.409211 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":123,"thread_start_us":138,"threads_started":1}
I20260806 12:23:47.410015 4642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb6dcb2d0844e22ac823d5ad7fc555b" candidate_uuid: "0b993543874840ec82a7d712c89e69a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba8f0498aca43808ccd934c640c247a" is_pre_election: true
I20260806 12:23:47.410027 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb6dcb2d0844e22ac823d5ad7fc555b" candidate_uuid: "0b993543874840ec82a7d712c89e69a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8248660b54416380a470eed2acda89" is_pre_election: true
I20260806 12:23:47.410313 4642 raft_consensus.cc:2454] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b993543874840ec82a7d712c89e69a3 in term 0.
I20260806 12:23:47.410313 4838 raft_consensus.cc:2454] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b993543874840ec82a7d712c89e69a3 in term 0.
I20260806 12:23:47.410805 4955 leader_election.cc:302] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b993543874840ec82a7d712c89e69a3, 1ba8f0498aca43808ccd934c640c247a, 5f8248660b54416380a470eed2acda89; no voters:
I20260806 12:23:47.411115 5176 raft_consensus.cc:2781] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:23:47.411206 5176 raft_consensus.cc:482] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:47.411254 5176 raft_consensus.cc:3037] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:47.412437 5176 raft_consensus.cc:504] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8248660b54416380a470eed2acda89" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41339 } } peers { permanent_uuid: "0b993543874840ec82a7d712c89e69a3" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 44195 } } peers { permanent_uuid: "1ba8f0498aca43808ccd934c640c247a" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 39239 } }
I20260806 12:23:47.412791 5176 leader_election.cc:288] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8248660b54416380a470eed2acda89 (127.28.210.66:41339), 1ba8f0498aca43808ccd934c640c247a (127.28.210.65:39239)
I20260806 12:23:47.413205 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb6dcb2d0844e22ac823d5ad7fc555b" candidate_uuid: "0b993543874840ec82a7d712c89e69a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8248660b54416380a470eed2acda89"
I20260806 12:23:47.413221 4642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb6dcb2d0844e22ac823d5ad7fc555b" candidate_uuid: "0b993543874840ec82a7d712c89e69a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba8f0498aca43808ccd934c640c247a"
I20260806 12:23:47.413339 4838 raft_consensus.cc:3037] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:47.413386 4642 raft_consensus.cc:3037] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:47.414388 4838 raft_consensus.cc:2454] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b993543874840ec82a7d712c89e69a3 in term 1.
I20260806 12:23:47.414566 4642 raft_consensus.cc:2454] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b993543874840ec82a7d712c89e69a3 in term 1.
I20260806 12:23:47.414726 4957 leader_election.cc:302] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b993543874840ec82a7d712c89e69a3, 5f8248660b54416380a470eed2acda89; no voters:
I20260806 12:23:47.414912 5176 raft_consensus.cc:2781] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:47.415156 5176 raft_consensus.cc:686] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0b993543874840ec82a7d712c89e69a3, State: Running, Role: LEADER
I20260806 12:23:47.415558 5176 consensus_queue.cc:237] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8248660b54416380a470eed2acda89" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41339 } } peers { permanent_uuid: "0b993543874840ec82a7d712c89e69a3" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 44195 } } peers { permanent_uuid: "1ba8f0498aca43808ccd934c640c247a" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 39239 } }
I20260806 12:23:47.419144 4402 catalog_manager.cc:5495] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b993543874840ec82a7d712c89e69a3 (127.28.210.67). New cstate: current_term: 1 leader_uuid: "0b993543874840ec82a7d712c89e69a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8248660b54416380a470eed2acda89" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b993543874840ec82a7d712c89e69a3" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 44195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ba8f0498aca43808ccd934c640c247a" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 39239 } health_report { overall_health: UNKNOWN } } }
W20260806 12:23:47.450784 4716 tablet.cc:2369] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:23:47.476555 5159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.474923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.28.210.67:44195)
0806 12:23:47.475072 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:47.475077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:47.475093 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:23:47.475295 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:47.475298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:47.475368 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:47.475444 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:47.476156 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:47.476160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:47.476163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:47.476177 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:47.476197 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:47.476205 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:47.476287 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:47.476357 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 12:23:47.476404 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 12:23:47.483630 4838 raft_consensus.cc:1261] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89 [term 1 FOLLOWER]: Refusing update from remote peer 0b993543874840ec82a7d712c89e69a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:47.483631 4642 raft_consensus.cc:1261] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a [term 1 FOLLOWER]: Refusing update from remote peer 0b993543874840ec82a7d712c89e69a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:47.484495 5176 consensus_queue.cc:1035] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ba8f0498aca43808ccd934c640c247a" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 39239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:47.484773 5178 consensus_queue.cc:1035] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8248660b54416380a470eed2acda89" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:47.493348 5182 mvcc.cc:205] Tried to move back new op lower bound from 7315533936563466240 to 7315533936297037824. Current Snapshot: MvccSnapshot[applied={T|T < 7315533936563466240}]
I20260806 12:23:47.494405 5184 mvcc.cc:205] Tried to move back new op lower bound from 7315533936563466240 to 7315533936297037824. Current Snapshot: MvccSnapshot[applied={T|T < 7315533936563466240}]
I20260806 12:23:47.495628 5183 mvcc.cc:205] Tried to move back new op lower bound from 7315533936563466240 to 7315533936297037824. Current Snapshot: MvccSnapshot[applied={T|T < 7315533936563466240}]
I20260806 12:23:47.505080 4402 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:47.505282 4402 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:47.507949 4402 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7cc2f3ba99f4fdb9a81a3abbf00c088: Sending DeleteTablet for 3 replicas of tablet eeb6dcb2d0844e22ac823d5ad7fc555b
I20260806 12:23:47.508982 5026 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb6dcb2d0844e22ac823d5ad7fc555b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:47 UTC) from {username='slave'} at 127.0.0.1:54262
I20260806 12:23:47.508980 4811 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb6dcb2d0844e22ac823d5ad7fc555b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:47 UTC) from {username='slave'} at 127.0.0.1:59304
I20260806 12:23:47.509032 4606 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb6dcb2d0844e22ac823d5ad7fc555b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:47 UTC) from {username='slave'} at 127.0.0.1:45172
I20260806 12:23:47.509822 5202 tablet_replica.cc:330] T eeb6dcb2d0844e22ac823d5ad7fc555b P 1ba8f0498aca43808ccd934c640c247a: stopping tablet replica
I20260806 12:23:47.509872 5201 tablet_replica.cc:330] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3: stopping tablet replica
I20260806 12:23:47.510269 5201 raft_consensus.cc:2229] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:23:47.510313 5198 tablet_replica.cc:330] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89: stopping tablet replica
I20260806 12:23:47.510619 5198 raft_consensus.cc:2229] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:47.510771 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 4511
I20260806 12:23:47.513667 5201 raft_consensus.cc:2258] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:47.514056 5198 raft_consensus.cc:2258] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:47.517324 5201 ts_tablet_manager.cc:1905] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:47.520876 5201 ts_tablet_manager.cc:1918] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:23:47.521005 5201 log.cc:1196] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/eeb6dcb2d0844e22ac823d5ad7fc555b
I20260806 12:23:47.521121 5198 ts_tablet_manager.cc:1905] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:47.521377 5201 ts_tablet_manager.cc:1939] T eeb6dcb2d0844e22ac823d5ad7fc555b P 0b993543874840ec82a7d712c89e69a3: Deleting consensus metadata
I20260806 12:23:47.524401 5198 ts_tablet_manager.cc:1918] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:23:47.524505 5198 log.cc:1196] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/eeb6dcb2d0844e22ac823d5ad7fc555b
I20260806 12:23:47.524750 5198 ts_tablet_manager.cc:1939] T eeb6dcb2d0844e22ac823d5ad7fc555b P 5f8248660b54416380a470eed2acda89: Deleting consensus metadata
I20260806 12:23:47.527554 4376 catalog_manager.cc:4841] TS 0b993543874840ec82a7d712c89e69a3 (127.28.210.67:44195): tablet eeb6dcb2d0844e22ac823d5ad7fc555b (table test-table [id=eb7d88ed04f44695b5127e58b7977f0e]) successfully deleted
I20260806 12:23:47.528640 4378 catalog_manager.cc:4841] TS 5f8248660b54416380a470eed2acda89 (127.28.210.66:41339): tablet eeb6dcb2d0844e22ac823d5ad7fc555b (table test-table [id=eb7d88ed04f44695b5127e58b7977f0e]) successfully deleted
W20260806 12:23:47.536366 4373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:39239 (error 108)
I20260806 12:23:47.536755 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 4719
I20260806 12:23:47.542002 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.541563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:39239 (local address 127.0.0.1:45180)
0806 12:23:47.541800 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:47.541885 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:39239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260806 12:23:47.542217 4373 catalog_manager.cc:4584] TS 1ba8f0498aca43808ccd934c640c247a (127.28.210.65:39239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eeb6dcb2d0844e22ac823d5ad7fc555b: Network error: Client connection negotiation failed: client connection to 127.28.210.65:39239: connect: Connection refused (error 111)
I20260806 12:23:47.558904 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 4932
I20260806 12:23:47.574826 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 4348
2026-08-06T12:23:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1388 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 12:23:47.609299 29513 mini_oidc.cc:152] Starting JWKS server
I20260806 12:23:47.611186 29513 webserver.cc:460] Webserver started at https://127.0.0.1:37337/ using document root <none> and password file <none>
I20260806 12:23:47.611347 29513 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 12:23:47.611603 29513 webserver.cc:460] Webserver started at http://127.0.0.1:33579/ using document root <none> and password file <none>
2026-08-06T12:23:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:47Z Disabled control of system clock
I20260806 12:23:47.625445 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:36543
--webserver_interface=127.28.210.126
--webserver_port=0
--builtin_ntp_servers=127.28.210.84:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:36543
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37337/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/testchainca.cert with env {}
W20260806 12:23:47.756419 5209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:47.756786 5209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:47.756836 5209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:47.761154 5209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:47.761255 5209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:47.761273 5209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:47.761293 5209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:47.761312 5209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 12:23:47.761330 5209 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 12:23:47.761346 5209 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37337/jwks.json
I20260806 12:23:47.767782 5209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:36543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:36543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37337/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:47.769219 5209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:47.770426 5209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:47.780401 5215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:47.780403 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:47.780403 5216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:47.780848 5209 server_base.cc:1034] running on GCE node
I20260806 12:23:47.781540 5209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:47.783017 5209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:47.784192 5209 hybrid_clock.cc:648] HybridClock initialized: now 1786019027784162 us; error 76 us; skew 500 ppm
I20260806 12:23:47.786580 5209 webserver.cc:460] Webserver started at http://127.28.210.126:43543/ using document root <none> and password file <none>
I20260806 12:23:47.787267 5209 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:47.787353 5209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:47.787580 5209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:47.789929 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "677402046b11444ab455e9578fee212f"
format_stamp: "Formatted at 2026-08-06 12:23:47 on dist-test-slave-lgr5"
server_key: "bb77b7cd01572d7524c4e16e6f6b369e"
server_key_iv: "0298ccf61d5fa4ddba416cae9da7addb"
server_key_version: "encryptionkey@0"
I20260806 12:23:47.790553 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "677402046b11444ab455e9578fee212f"
format_stamp: "Formatted at 2026-08-06 12:23:47 on dist-test-slave-lgr5"
server_key: "bb77b7cd01572d7524c4e16e6f6b369e"
server_key_iv: "0298ccf61d5fa4ddba416cae9da7addb"
server_key_version: "encryptionkey@0"
I20260806 12:23:47.795140 5209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 12:23:47.798173 5227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:47.799649 5209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:23:47.799806 5209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "677402046b11444ab455e9578fee212f"
format_stamp: "Formatted at 2026-08-06 12:23:47 on dist-test-slave-lgr5"
server_key: "bb77b7cd01572d7524c4e16e6f6b369e"
server_key_iv: "0298ccf61d5fa4ddba416cae9da7addb"
server_key_version: "encryptionkey@0"
I20260806 12:23:47.799952 5209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:47.827471 5209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:47.828380 5209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:47.828589 5209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:47.837278 5209 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:36543
I20260806 12:23:47.837368 5321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:36543 every 8 connection(s)
I20260806 12:23:47.838436 5209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:47.842041 5322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:47.844523 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 5209
I20260806 12:23:47.844671 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:47.844992 29513 external_mini_cluster.cc:1428] Setting key 915d9de72b7d075f0eeecb4445411cb4
I20260806 12:23:47.849196 5322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f: Bootstrap starting.
I20260806 12:23:47.851804 5322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:47.852846 5322 log.cc:824] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:47.855645 5322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f: No bootstrap required, opened a new log
I20260806 12:23:47.855847 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.846691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.28.210.126:36543)
0806 12:23:47.847416 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:47.847433 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:47.847498 (+ 65us) server_negotiation.cc:407] Connection header received
0806 12:23:47.848234 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:47.848256 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:47.848794 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:47.849154 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:47.849671 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:47.850690 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:47.851621 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:47.852275 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:47.852576 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:47.853547 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:47.853553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:47.853569 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:47.854063 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:47.854141 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:47.854170 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:47.854278 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:47.854485 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 12:23:47.854801 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":149,"threads_started":1}
I20260806 12:23:47.859105 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:23:47.859838 5322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677402046b11444ab455e9578fee212f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36543 } }
I20260806 12:23:47.860222 5322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:47.860291 5322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 677402046b11444ab455e9578fee212f, State: Initialized, Role: FOLLOWER
I20260806 12:23:47.860775 5322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677402046b11444ab455e9578fee212f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36543 } }
I20260806 12:23:47.860961 5322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:47.861037 5322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:47.861146 5322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:47.862214 5322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677402046b11444ab455e9578fee212f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36543 } }
I20260806 12:23:47.862589 5322 leader_election.cc:302] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 677402046b11444ab455e9578fee212f; no voters:
I20260806 12:23:47.862958 5322 leader_election.cc:288] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:47.863087 5331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:47.863350 5331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [term 1 LEADER]: Becoming Leader. State: Replica: 677402046b11444ab455e9578fee212f, State: Running, Role: LEADER
I20260806 12:23:47.863783 5331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677402046b11444ab455e9578fee212f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36543 } }
I20260806 12:23:47.864418 5322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:47.865701 5332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "677402046b11444ab455e9578fee212f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677402046b11444ab455e9578fee212f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36543 } } }
I20260806 12:23:47.865887 5332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:47.866194 5333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 677402046b11444ab455e9578fee212f. Latest consensus state: current_term: 1 leader_uuid: "677402046b11444ab455e9578fee212f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677402046b11444ab455e9578fee212f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36543 } } }
I20260806 12:23:47.866302 5333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:47.866647 5351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:47.870265 5351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:47.877163 5351 catalog_manager.cc:1347] Generated new cluster ID: 422e5ef0538246349d8af83815109092
I20260806 12:23:47.877269 5351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:47.890141 5351 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:47.891233 5351 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:47.898049 5351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 677402046b11444ab455e9578fee212f: Generated new TSK 0
I20260806 12:23:47.899021 5351 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 12:23:47.943590 5370 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 12:23:47.943764 5370 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 12:23:47.944751 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:47.922584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.28.210.126:36543)
0806 12:23:47.922730 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:47.922736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:47.923266 (+ 530us) server_negotiation.cc:407] Connection header received
0806 12:23:47.923776 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:47.923780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:47.923928 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:47.924054 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0806 12:23:47.924433 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:47.925293 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:47.926608 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:47.926970 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:47.927079 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:47.927575 (+ 496us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 12:23:47.944198 (+ 16623us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 12:23:47.944578 (+ 380us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 12:23:47.944859 5327 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 12:23:47.946273 5369 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:23:47.922576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36543 (local address 127.0.0.1:56166)
0806 12:23:47.923146 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:47.923165 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:47.923405 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:47.924130 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:47.924134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:47.924145 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0806 12:23:47.924296 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:47.924304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:47.925429 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:47.925433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:47.926457 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:47.926470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:47.927126 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:47.927131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:47.927236 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:47.927312 (+ 76us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 12:23:47.944505 (+ 17193us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 12:23:47.944762 (+ 257us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.210.126:36543: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":250,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.210.126:36543: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 12:23:47.947082 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 5209
2026-08-06T12:23:47Z chronyd exiting
I20260806 12:23:48.012866 29513 test_util.cc:175] -----------------------------------------------
I20260806 12:23:48.012979 29513 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786018995816160-29513-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (405 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 12:23:48.014288 29513 mini_oidc.cc:152] Starting JWKS server
I20260806 12:23:48.015295 29513 webserver.cc:460] Webserver started at https://127.0.0.1:39247/ using document root <none> and password file <none>
I20260806 12:23:48.015405 29513 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 12:23:48.015707 29513 webserver.cc:460] Webserver started at http://127.0.0.1:44221/ using document root <none> and password file <none>
2026-08-06T12:23:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:48Z Disabled control of system clock
I20260806 12:23:48.027069 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:46601
--webserver_interface=127.28.210.126
--webserver_port=0
--builtin_ntp_servers=127.28.210.84:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:46601
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39247/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/testchainca.cert with env {}
W20260806 12:23:48.163652 5380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:48.164021 5380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:48.164067 5380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:48.168629 5380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:48.168730 5380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:48.168744 5380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:48.168762 5380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:48.168777 5380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 12:23:48.168793 5380 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 12:23:48.168807 5380 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39247/jwks.json
I20260806 12:23:48.175377 5380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:46601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:46601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39247/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:48.176718 5380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:48.177830 5380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:48.187937 5388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:48.188035 5380 server_base.cc:1034] running on GCE node
W20260806 12:23:48.187963 5390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:48.187963 5393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:48.188848 5380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:48.190322 5380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:48.191557 5380 hybrid_clock.cc:648] HybridClock initialized: now 1786019028191498 us; error 79 us; skew 500 ppm
I20260806 12:23:48.200407 5380 webserver.cc:460] Webserver started at http://127.28.210.126:34435/ using document root <none> and password file <none>
I20260806 12:23:48.201244 5380 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:48.201357 5380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:48.201635 5380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:48.204713 5380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "0762ab1cb8f84ceeb418461aae2b316e"
format_stamp: "Formatted at 2026-08-06 12:23:48 on dist-test-slave-lgr5"
server_key: "9884630819ca51f48472f764fd0e37ee"
server_key_iv: "405b91242b1d26481593cdec6d5c1742"
server_key_version: "encryptionkey@0"
I20260806 12:23:48.205511 5380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "0762ab1cb8f84ceeb418461aae2b316e"
format_stamp: "Formatted at 2026-08-06 12:23:48 on dist-test-slave-lgr5"
server_key: "9884630819ca51f48472f764fd0e37ee"
server_key_iv: "405b91242b1d26481593cdec6d5c1742"
server_key_version: "encryptionkey@0"
I20260806 12:23:48.210572 5380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 12:23:48.214849 5405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:48.216881 5380 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:23:48.217078 5380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "0762ab1cb8f84ceeb418461aae2b316e"
format_stamp: "Formatted at 2026-08-06 12:23:48 on dist-test-slave-lgr5"
server_key: "9884630819ca51f48472f764fd0e37ee"
server_key_iv: "405b91242b1d26481593cdec6d5c1742"
server_key_version: "encryptionkey@0"
I20260806 12:23:48.217244 5380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:48.238881 5380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:48.239866 5380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:48.240161 5380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:48.248891 5380 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:46601
I20260806 12:23:48.249046 5488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:46601 every 8 connection(s)
I20260806 12:23:48.250102 5380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:48.254114 5491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:48.256572 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 5380
I20260806 12:23:48.256721 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:48.257078 29513 external_mini_cluster.cc:1428] Setting key b2ae492233e07bdeae58dd4ed7241dc4
I20260806 12:23:48.262609 5491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e: Bootstrap starting.
I20260806 12:23:48.265646 5491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:48.266690 5491 log.cc:824] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:48.268508 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:48.258783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.28.210.126:46601)
0806 12:23:48.259638 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:48.259659 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:48.259741 (+ 82us) server_negotiation.cc:407] Connection header received
0806 12:23:48.260603 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:48.260626 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:48.261146 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:48.261598 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:48.262129 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.263306 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.264423 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.265012 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.265186 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:48.266231 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:48.266237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:48.266254 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:48.266744 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:48.266827 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:48.266861 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:48.266967 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:48.267125 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 12:23:48.267449 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":196,"threads_started":1}
I20260806 12:23:48.269191 5491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e: No bootstrap required, opened a new log
I20260806 12:23:48.271907 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:23:48.273622 5491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0762ab1cb8f84ceeb418461aae2b316e" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46601 } }
I20260806 12:23:48.273910 5491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:48.273974 5491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0762ab1cb8f84ceeb418461aae2b316e, State: Initialized, Role: FOLLOWER
I20260806 12:23:48.274479 5491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0762ab1cb8f84ceeb418461aae2b316e" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46601 } }
I20260806 12:23:48.274641 5491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:48.274717 5491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:48.274829 5491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:48.275975 5491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0762ab1cb8f84ceeb418461aae2b316e" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46601 } }
I20260806 12:23:48.276403 5491 leader_election.cc:302] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0762ab1cb8f84ceeb418461aae2b316e; no voters:
I20260806 12:23:48.276813 5491 leader_election.cc:288] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:48.276996 5500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:48.277258 5500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [term 1 LEADER]: Becoming Leader. State: Replica: 0762ab1cb8f84ceeb418461aae2b316e, State: Running, Role: LEADER
I20260806 12:23:48.277691 5500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0762ab1cb8f84ceeb418461aae2b316e" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46601 } }
I20260806 12:23:48.278431 5491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:48.280949 5504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0762ab1cb8f84ceeb418461aae2b316e. Latest consensus state: current_term: 1 leader_uuid: "0762ab1cb8f84ceeb418461aae2b316e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0762ab1cb8f84ceeb418461aae2b316e" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46601 } } }
I20260806 12:23:48.281193 5504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:48.282135 5501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0762ab1cb8f84ceeb418461aae2b316e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0762ab1cb8f84ceeb418461aae2b316e" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46601 } } }
I20260806 12:23:48.282270 5501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e [sys.catalog]: This master's current role is: LEADER
W20260806 12:23:48.285166 5528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:23:48.285269 5528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:23:48.285358 5532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:48.286185 5532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:48.292636 5532 catalog_manager.cc:1347] Generated new cluster ID: a94190509aec4e18bfd5c4ec7fe74625
I20260806 12:23:48.292744 5532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:48.308347 5532 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:48.309545 5532 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:48.320302 5532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0762ab1cb8f84ceeb418461aae2b316e: Generated new TSK 0
I20260806 12:23:48.321727 5532 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 12:23:48.363749 5547 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 12:23:48.363902 5547 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 12:23:48.364634 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:48.338106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36628 (local address 127.28.210.126:46601)
0806 12:23:48.338249 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:48.338254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:48.338502 (+ 248us) server_negotiation.cc:407] Connection header received
0806 12:23:48.338757 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:48.338761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:48.338828 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:48.338951 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0806 12:23:48.339263 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.339846 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.341234 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.341482 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.341593 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:48.342032 (+ 439us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 12:23:48.364190 (+ 22158us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 12:23:48.364505 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 12:23:48.364701 5498 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 12:23:48.364907 5546 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:23:48.337929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:46601 (local address 127.0.0.1:36628)
0806 12:23:48.338350 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:48.338370 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:48.338529 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:48.338969 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:48.338972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:48.338982 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0806 12:23:48.339141 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:48.339148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.339970 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.339974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:48.341060 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:48.341076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.341633 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.341637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:48.341743 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:48.341793 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 12:23:48.364456 (+ 22663us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 12:23:48.364674 (+ 218us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.210.126:46601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":131,"threads_started":1}
I20260806 12:23:48.365788 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 5380
2026-08-06T12:23:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (407 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 12:23:48.421468 29513 mini_oidc.cc:152] Starting JWKS server
I20260806 12:23:48.422402 29513 webserver.cc:460] Webserver started at https://127.0.0.1:33087/ using document root <none> and password file <none>
I20260806 12:23:48.422500 29513 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 12:23:48.422763 29513 webserver.cc:460] Webserver started at http://127.0.0.1:35571/ using document root <none> and password file <none>
2026-08-06T12:23:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:48Z Disabled control of system clock
I20260806 12:23:48.434170 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:35561
--webserver_interface=127.28.210.126
--webserver_port=0
--builtin_ntp_servers=127.28.210.84:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:35561
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33087/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 12:23:48.559440 5557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:48.559959 5557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:48.560035 5557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:48.564548 5557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:48.564642 5557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:48.564667 5557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:48.564689 5557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:48.564711 5557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 12:23:48.564734 5557 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 12:23:48.564761 5557 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33087/jwks.json
I20260806 12:23:48.570118 5557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:35561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:35561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33087/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:48.571744 5557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:48.573380 5557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:48.583729 5565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:48.583734 5567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:48.583729 5576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:48.583804 5557 server_base.cc:1034] running on GCE node
I20260806 12:23:48.584900 5557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:48.586416 5557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:48.587632 5557 hybrid_clock.cc:648] HybridClock initialized: now 1786019028587613 us; error 44 us; skew 500 ppm
I20260806 12:23:48.590098 5557 webserver.cc:460] Webserver started at http://127.28.210.126:40679/ using document root <none> and password file <none>
I20260806 12:23:48.590777 5557 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:48.590857 5557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:48.591102 5557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:48.593799 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "e23e3e9eb3314ac8b18d917ecc50ced6"
format_stamp: "Formatted at 2026-08-06 12:23:48 on dist-test-slave-lgr5"
server_key: "2ae273ecae3f845fbf02c5dbda19d617"
server_key_iv: "2c5cb42752b456ce4df7336f841593ac"
server_key_version: "encryptionkey@0"
I20260806 12:23:48.594478 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "e23e3e9eb3314ac8b18d917ecc50ced6"
format_stamp: "Formatted at 2026-08-06 12:23:48 on dist-test-slave-lgr5"
server_key: "2ae273ecae3f845fbf02c5dbda19d617"
server_key_iv: "2c5cb42752b456ce4df7336f841593ac"
server_key_version: "encryptionkey@0"
I20260806 12:23:48.599272 5557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 12:23:48.602583 5584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:48.604223 5557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:48.604429 5557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "e23e3e9eb3314ac8b18d917ecc50ced6"
format_stamp: "Formatted at 2026-08-06 12:23:48 on dist-test-slave-lgr5"
server_key: "2ae273ecae3f845fbf02c5dbda19d617"
server_key_iv: "2c5cb42752b456ce4df7336f841593ac"
server_key_version: "encryptionkey@0"
I20260806 12:23:48.604583 5557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:48.617622 5557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:48.618544 5557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:48.618762 5557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:48.626619 5557 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:35561
I20260806 12:23:48.626672 5663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:35561 every 8 connection(s)
I20260806 12:23:48.627709 5557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:48.631521 5664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:48.632375 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 5557
I20260806 12:23:48.632500 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:48.632779 29513 external_mini_cluster.cc:1428] Setting key 00c859c68415ae759528eff1f033fc3d
I20260806 12:23:48.638511 5664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6: Bootstrap starting.
I20260806 12:23:48.641405 5664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:48.642386 5664 log.cc:824] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:48.642858 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:48.634362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.28.210.126:35561)
0806 12:23:48.635102 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:48.635119 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:48.635159 (+ 40us) server_negotiation.cc:407] Connection header received
0806 12:23:48.635843 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:48.635862 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:48.636348 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:48.636678 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:48.637127 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.638113 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.639146 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.639710 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.639885 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:48.640871 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:48.640877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:48.640892 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:48.641345 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:48.641408 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:48.641430 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:48.641523 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:48.641677 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 12:23:48.641985 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":141,"threads_started":1}
I20260806 12:23:48.645012 5664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6: No bootstrap required, opened a new log
I20260806 12:23:48.646279 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:23:48.649626 5664 raft_consensus.cc:348] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23e3e9eb3314ac8b18d917ecc50ced6" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35561 } }
I20260806 12:23:48.649919 5664 raft_consensus.cc:374] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:48.649972 5664 raft_consensus.cc:729] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23e3e9eb3314ac8b18d917ecc50ced6, State: Initialized, Role: FOLLOWER
I20260806 12:23:48.650394 5664 consensus_queue.cc:260] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23e3e9eb3314ac8b18d917ecc50ced6" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35561 } }
I20260806 12:23:48.650522 5664 raft_consensus.cc:388] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:48.650583 5664 raft_consensus.cc:482] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:48.650651 5664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:48.651700 5664 raft_consensus.cc:504] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23e3e9eb3314ac8b18d917ecc50ced6" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35561 } }
I20260806 12:23:48.652107 5664 leader_election.cc:302] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e23e3e9eb3314ac8b18d917ecc50ced6; no voters:
I20260806 12:23:48.652467 5664 leader_election.cc:288] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:48.652720 5671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:48.652993 5671 raft_consensus.cc:686] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [term 1 LEADER]: Becoming Leader. State: Replica: e23e3e9eb3314ac8b18d917ecc50ced6, State: Running, Role: LEADER
I20260806 12:23:48.653353 5671 consensus_queue.cc:237] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23e3e9eb3314ac8b18d917ecc50ced6" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35561 } }
I20260806 12:23:48.653878 5664 sys_catalog.cc:564] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:48.655279 5674 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e23e3e9eb3314ac8b18d917ecc50ced6. Latest consensus state: current_term: 1 leader_uuid: "e23e3e9eb3314ac8b18d917ecc50ced6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23e3e9eb3314ac8b18d917ecc50ced6" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35561 } } }
I20260806 12:23:48.655337 5673 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e23e3e9eb3314ac8b18d917ecc50ced6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23e3e9eb3314ac8b18d917ecc50ced6" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35561 } } }
I20260806 12:23:48.655409 5674 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:48.655474 5673 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:48.655858 5683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:48.659827 5683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:48.667320 5683 catalog_manager.cc:1347] Generated new cluster ID: ae1f9377a73a43d2abc0e7018cfac2be
I20260806 12:23:48.667438 5683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:48.678032 5683 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:48.679533 5683 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:48.689097 5683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e23e3e9eb3314ac8b18d917ecc50ced6: Generated new TSK 0
I20260806 12:23:48.689942 5683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:23:48.707643 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:48.702664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.28.210.126:35561)
0806 12:23:48.702810 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:48.702815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:48.702974 (+ 159us) server_negotiation.cc:407] Connection header received
0806 12:23:48.703155 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:48.703157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:48.703208 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:48.703300 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0806 12:23:48.703531 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.704110 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.705352 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.705568 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.705634 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:48.705965 (+ 331us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 12:23:48.707245 (+ 1280us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33087/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 12:23:48.707499 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53008: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33087/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 12:23:48.707724 5669 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53008: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33087/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 12:23:48.710466 5713 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:23:48.702576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35561 (local address 127.0.0.1:53008)
0806 12:23:48.702897 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:48.702913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:48.703033 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:48.703313 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:48.703316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:48.703324 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 12:23:48.703448 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:48.703454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.704224 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.704228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:48.705239 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:48.705251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.705674 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.705679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:48.705770 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:48.705800 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 12:23:48.707414 (+ 1614us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33087/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 12:23:48.710312 (+ 2898us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.210.126:35561: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33087/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":2678,"thread_start_us":97,"threads_started":1}
I20260806 12:23:48.711163 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 5557
E20260806 12:23:48.729434 5714 webserver.cc:533] Webserver: sslize failed:
E20260806 12:23:48.729524 5714 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T12:23:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (341 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T12:23:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:48Z Disabled control of system clock
I20260806 12:23:48.773851 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:43213
--webserver_interface=127.28.210.126
--webserver_port=0
--builtin_ntp_servers=127.28.210.84:44803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:43213
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/testchainca.cert with env {}
W20260806 12:23:48.898026 5720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:48.898389 5720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:48.898454 5720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:48.902902 5720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:48.903000 5720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:48.903024 5720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:48.903046 5720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:48.903067 5720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 12:23:48.903097 5720 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 12:23:48.903126 5720 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 12:23:48.909192 5720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:43213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:43213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:48.910480 5720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:48.911612 5720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:23:48.921957 5720 server_base.cc:1034] running on GCE node
W20260806 12:23:48.921897 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:48.921909 5730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:48.921903 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:48.922943 5720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:48.924378 5720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:48.925582 5720 hybrid_clock.cc:648] HybridClock initialized: now 1786019028925553 us; error 47 us; skew 500 ppm
I20260806 12:23:48.927956 5720 webserver.cc:460] Webserver started at http://127.28.210.126:33067/ using document root <none> and password file <none>
I20260806 12:23:48.928756 5720 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:48.928846 5720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:48.929114 5720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:48.932308 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "edf16ab8754543ed95503e031ae742a6"
format_stamp: "Formatted at 2026-08-06 12:23:48 on dist-test-slave-lgr5"
server_key: "eebc14f49cac2260006094ddbed973c6"
server_key_iv: "7986301ae14be7458b7d776affc4d700"
server_key_version: "encryptionkey@0"
I20260806 12:23:48.933535 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "edf16ab8754543ed95503e031ae742a6"
format_stamp: "Formatted at 2026-08-06 12:23:48 on dist-test-slave-lgr5"
server_key: "eebc14f49cac2260006094ddbed973c6"
server_key_iv: "7986301ae14be7458b7d776affc4d700"
server_key_version: "encryptionkey@0"
I20260806 12:23:48.940222 5720 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260806 12:23:48.943486 5742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:48.945123 5720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:48.945300 5720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "edf16ab8754543ed95503e031ae742a6"
format_stamp: "Formatted at 2026-08-06 12:23:48 on dist-test-slave-lgr5"
server_key: "eebc14f49cac2260006094ddbed973c6"
server_key_iv: "7986301ae14be7458b7d776affc4d700"
server_key_version: "encryptionkey@0"
I20260806 12:23:48.945446 5720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:48.961076 5720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:48.961956 5720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:48.962149 5720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:48.973140 5720 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:43213
I20260806 12:23:48.973275 5829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:43213 every 8 connection(s)
I20260806 12:23:48.974817 5720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:48.979156 5830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:48.982594 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 5720
I20260806 12:23:48.982725 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:48.983022 29513 external_mini_cluster.cc:1428] Setting key c4963edeb686084a2a4abef794f359ec
I20260806 12:23:48.986800 5830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6: Bootstrap starting.
I20260806 12:23:48.989969 5830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:48.991127 5830 log.cc:824] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:48.994421 5830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6: No bootstrap required, opened a new log
I20260806 12:23:48.996497 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:48.985149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.28.210.126:43213)
0806 12:23:48.986070 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:48.986089 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:48.986139 (+ 50us) server_negotiation.cc:407] Connection header received
0806 12:23:48.986870 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:48.986891 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:48.987414 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:48.987758 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:48.988464 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.990044 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.991540 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:48.992459 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:48.992664 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:48.993896 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:48.993903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:48.993920 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:23:48.994465 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:48.994560 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:23:48.994596 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:48.994742 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:48.994941 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 12:23:48.995371 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":183,"threads_started":1}
I20260806 12:23:48.999059 5830 raft_consensus.cc:348] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf16ab8754543ed95503e031ae742a6" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43213 } }
I20260806 12:23:48.999431 5830 raft_consensus.cc:374] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:48.999538 5830 raft_consensus.cc:729] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf16ab8754543ed95503e031ae742a6, State: Initialized, Role: FOLLOWER
I20260806 12:23:49.000257 5830 consensus_queue.cc:260] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf16ab8754543ed95503e031ae742a6" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43213 } }
I20260806 12:23:49.000509 5830 raft_consensus.cc:388] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:49.000593 5830 raft_consensus.cc:482] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:49.000720 5830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:49.001103 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:23:49.002310 5830 raft_consensus.cc:504] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf16ab8754543ed95503e031ae742a6" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43213 } }
I20260806 12:23:49.002776 5830 leader_election.cc:302] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edf16ab8754543ed95503e031ae742a6; no voters:
I20260806 12:23:49.003185 5830 leader_election.cc:288] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:49.003309 5836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:49.003527 5836 raft_consensus.cc:686] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [term 1 LEADER]: Becoming Leader. State: Replica: edf16ab8754543ed95503e031ae742a6, State: Running, Role: LEADER
I20260806 12:23:49.003948 5836 consensus_queue.cc:237] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf16ab8754543ed95503e031ae742a6" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43213 } }
I20260806 12:23:49.004997 5830 sys_catalog.cc:564] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:49.006108 5837 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edf16ab8754543ed95503e031ae742a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf16ab8754543ed95503e031ae742a6" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43213 } } }
I20260806 12:23:49.006153 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edf16ab8754543ed95503e031ae742a6. Latest consensus state: current_term: 1 leader_uuid: "edf16ab8754543ed95503e031ae742a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf16ab8754543ed95503e031ae742a6" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43213 } } }
I20260806 12:23:49.006269 5837 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:49.006269 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6 [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:49.006865 5847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:49.010959 5847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:49.018728 5847 catalog_manager.cc:1347] Generated new cluster ID: b84cd5d30dd44a74aa4c842b88da7f7f
I20260806 12:23:49.018867 5847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:49.033098 5847 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:49.034372 5847 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:49.045265 5847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edf16ab8754543ed95503e031ae742a6: Generated new TSK 0
I20260806 12:23:49.046231 5847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:23:49.538296 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:49.063561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.28.210.126:43213)
0806 12:23:49.063726 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:49.063732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:49.064062 (+ 330us) server_negotiation.cc:407] Connection header received
0806 12:23:49.064318 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:49.064321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:49.064423 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:49.064540 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0806 12:23:49.064886 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:49.065535 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:49.066987 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:49.067271 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:49.067453 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:49.068254 (+ 801us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 12:23:49.537690 (+469436us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 12:23:49.538121 (+ 431us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42806: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 12:23:49.538394 5834 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42806: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 12:23:49.538787 5873 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:23:49.063352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:43213 (local address 127.0.0.1:42806)
0806 12:23:49.063903 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:49.063923 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:49.064117 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:49.064564 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:49.064568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:49.064577 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 12:23:49.064731 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:49.064739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:49.065676 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:49.065681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:49.066773 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:49.066789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:49.067525 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:49.067530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:49.067652 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:49.067700 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 12:23:49.538118 (+470418us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 12:23:49.538504 (+ 386us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.210.126:43213: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":157,"threads_started":1}
I20260806 12:23:49.539549 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 5720
2026-08-06T12:23:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (808 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (146 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (144 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:23:49 dist-test-slave-lgr5 krb5kdc[5894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:23:49 dist-test-slave-lgr5 krb5kdc[5894](info): set up 2 sockets
Aug 06 12:23:49 dist-test-slave-lgr5 krb5kdc[5894](info): commencing operation
krb5kdc: starting...
W20260806 12:23:51.050621 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:23:51 dist-test-slave-lgr5 krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:23:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:23:51.208560 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:43531
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:43531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:51.339846 5914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:51.340421 5914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:51.340538 5914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:51.345465 5914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:51.345571 5914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:51.345605 5914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:51.345628 5914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:51.345649 5914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:51.351656 5914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:43531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:43531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:51.352955 5914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:51.354086 5914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:51.364108 5929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:51.364352 5914 server_base.cc:1034] running on GCE node
W20260806 12:23:51.364108 5926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:51.364104 5925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:51.365170 5914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:51.366633 5914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:51.367854 5914 hybrid_clock.cc:648] HybridClock initialized: now 1786019031367836 us; error 37 us; skew 500 ppm
Aug 06 12:23:51 dist-test-slave-lgr5 krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019031, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:51.371119 5914 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:23:51.372225 5914 webserver.cc:460] Webserver started at http://127.28.210.126:33413/ using document root <none> and password file <none>
I20260806 12:23:51.372830 5914 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:51.372905 5914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:51.373140 5914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:51.375480 5914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "186205d06e4149ebb375cad2f1819d1e"
format_stamp: "Formatted at 2026-08-06 12:23:51 on dist-test-slave-lgr5"
server_key: "d80c2857fd382292706aacddf60ef5fc"
server_key_iv: "4a993a2b2588fc0e88ae793733866bcf"
server_key_version: "encryptionkey@0"
I20260806 12:23:51.376163 5914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "186205d06e4149ebb375cad2f1819d1e"
format_stamp: "Formatted at 2026-08-06 12:23:51 on dist-test-slave-lgr5"
server_key: "d80c2857fd382292706aacddf60ef5fc"
server_key_iv: "4a993a2b2588fc0e88ae793733866bcf"
server_key_version: "encryptionkey@0"
I20260806 12:23:51.380951 5914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 12:23:51.384186 5938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:51.385924 5914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 12:23:51.386113 5914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "186205d06e4149ebb375cad2f1819d1e"
format_stamp: "Formatted at 2026-08-06 12:23:51 on dist-test-slave-lgr5"
server_key: "d80c2857fd382292706aacddf60ef5fc"
server_key_iv: "4a993a2b2588fc0e88ae793733866bcf"
server_key_version: "encryptionkey@0"
I20260806 12:23:51.386286 5914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:51.404525 5914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:51.408519 5914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:51.408767 5914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:51.418036 5914 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:43531
I20260806 12:23:51.418493 6025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:43531 every 8 connection(s)
I20260806 12:23:51.419565 5914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:51.423375 6026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:51.428625 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 5914
I20260806 12:23:51.428751 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:51.429040 29513 external_mini_cluster.cc:1428] Setting key f226027dd71208b85a4086f7dc24dfd6
I20260806 12:23:51.430578 6026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e: Bootstrap starting.
I20260806 12:23:51.433283 6026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:51.434149 6026 log.cc:824] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:51.436419 6026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e: No bootstrap required, opened a new log
Aug 06 12:23:51 dist-test-slave-lgr5 krb5kdc[5894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:51.440615 6026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186205d06e4149ebb375cad2f1819d1e" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43531 } }
I20260806 12:23:51.440922 6026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:51.440986 6026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186205d06e4149ebb375cad2f1819d1e, State: Initialized, Role: FOLLOWER
I20260806 12:23:51.441473 6026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186205d06e4149ebb375cad2f1819d1e" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43531 } }
I20260806 12:23:51.441622 6026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:51.441695 6026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:51.441807 6026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:51.443044 6026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186205d06e4149ebb375cad2f1819d1e" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43531 } }
I20260806 12:23:51.443485 6026 leader_election.cc:302] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 186205d06e4149ebb375cad2f1819d1e; no voters:
I20260806 12:23:51.443854 6026 leader_election.cc:288] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:51.443953 6032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:51.444217 6032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [term 1 LEADER]: Becoming Leader. State: Replica: 186205d06e4149ebb375cad2f1819d1e, State: Running, Role: LEADER
I20260806 12:23:51.444604 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:51.430652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.28.210.126:43531)
0806 12:23:51.431192 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:51.431209 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:51.431253 (+ 44us) server_negotiation.cc:407] Connection header received
0806 12:23:51.432011 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:51.432035 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:51.432479 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:51.432833 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:51.433726 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:51.434815 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:51.435720 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:51.436285 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:51.436443 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:51.439088 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:51.439094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:51.439110 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:51.439144 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:51.440989 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:51.441560 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:51.441567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:51.441572 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:51.441728 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:51.442091 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:51.442094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:51.442095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:51.442555 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:51.442719 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:51.443089 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 12:23:51.443452 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":166,"threads_started":1}
I20260806 12:23:51.444633 6032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186205d06e4149ebb375cad2f1819d1e" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43531 } }
I20260806 12:23:51.445286 6026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:51.445945 6032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 186205d06e4149ebb375cad2f1819d1e. Latest consensus state: current_term: 1 leader_uuid: "186205d06e4149ebb375cad2f1819d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186205d06e4149ebb375cad2f1819d1e" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43531 } } }
I20260806 12:23:51.446076 6032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:51.446652 6033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "186205d06e4149ebb375cad2f1819d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186205d06e4149ebb375cad2f1819d1e" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43531 } } }
I20260806 12:23:51.446843 6033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:51.447142 6045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:51.450788 6045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:51.457482 6045 catalog_manager.cc:1347] Generated new cluster ID: f90f4eddb67343de8c1d826c29997aa8
I20260806 12:23:51.457571 6045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:51.467664 6045 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:51.468839 6045 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:51.478675 6045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 186205d06e4149ebb375cad2f1819d1e: Generated new TSK 0
I20260806 12:23:51.479533 6045 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:23:51.516636 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:43531
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:51.653843 6066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:51.654193 6066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:51.654244 6066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:51.658887 6066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:51.658998 6066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:51.659066 6066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:23:51.665046 6066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:43531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:51.666361 6066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:51.667559 6066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:51.678783 6074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:51.678771 6076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:51.678929 6072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:51.679448 6066 server_base.cc:1034] running on GCE node
I20260806 12:23:51.679955 6066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:51.680862 6066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:51.682094 6066 hybrid_clock.cc:648] HybridClock initialized: now 1786019031682069 us; error 40 us; skew 500 ppm
Aug 06 12:23:51 dist-test-slave-lgr5 krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019031, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:51.686304 6066 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:23:51.687618 6066 webserver.cc:460] Webserver started at http://127.28.210.65:39789/ using document root <none> and password file <none>
I20260806 12:23:51.688445 6066 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:51.688534 6066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:51.688746 6066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:51.691694 6066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "c920f906d56c44ee825b7726cbfa5388"
format_stamp: "Formatted at 2026-08-06 12:23:51 on dist-test-slave-lgr5"
server_key: "84a280de1e98c2a0fb3b9bfbfd83eb71"
server_key_iv: "35a6b33ea1f50537a3ac5d3ecfaf582e"
server_key_version: "encryptionkey@0"
I20260806 12:23:51.692538 6066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "c920f906d56c44ee825b7726cbfa5388"
format_stamp: "Formatted at 2026-08-06 12:23:51 on dist-test-slave-lgr5"
server_key: "84a280de1e98c2a0fb3b9bfbfd83eb71"
server_key_iv: "35a6b33ea1f50537a3ac5d3ecfaf582e"
server_key_version: "encryptionkey@0"
I20260806 12:23:51.697113 6066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 12:23:51.700052 6090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:51.701567 6066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 12:23:51.701697 6066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "c920f906d56c44ee825b7726cbfa5388"
format_stamp: "Formatted at 2026-08-06 12:23:51 on dist-test-slave-lgr5"
server_key: "84a280de1e98c2a0fb3b9bfbfd83eb71"
server_key_iv: "35a6b33ea1f50537a3ac5d3ecfaf582e"
server_key_version: "encryptionkey@0"
I20260806 12:23:51.701848 6066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:51.730149 6066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:51.733633 6066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:51.733834 6066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:51.734526 6066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:51.735455 6066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:51.735505 6066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:51.735550 6066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:51.735579 6066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:51.744935 6066 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:32881
I20260806 12:23:51.744987 6267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:32881 every 8 connection(s)
I20260806 12:23:51.746098 6066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:23:51.746496 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 6066
I20260806 12:23:51.746614 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:23:51.746966 29513 external_mini_cluster.cc:1428] Setting key ae88aaf434b2e88ad111b1d1d7a9c15b
Aug 06 12:23:51 dist-test-slave-lgr5 krb5kdc[5894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019031, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:23:51.764143 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:51.748210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:41677 (local address 127.28.210.126:43531)
0806 12:23:51.748397 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:51.748400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:51.749625 (+ 1225us) server_negotiation.cc:407] Connection header received
0806 12:23:51.751691 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:51.751695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:51.751760 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:51.751840 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:51.753572 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:51.754367 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:51.755832 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:51.756292 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:51.756440 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:51.759544 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:51.759548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:51.759550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:51.759581 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:51.761315 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:51.761984 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:51.761989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:51.761990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:51.762081 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:51.762638 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:51.762641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:51.762642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:51.762891 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:51.762986 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:51.763756 (+ 770us) server_negotiation.cc:299] Negotiation successful
0806 12:23:51.763928 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 12:23:51.765130 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:51.748805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:43531 (local address 127.28.210.65:41677)
0806 12:23:51.749432 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:51.749502 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:51.751424 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:51.752065 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:51.752073 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:51.752708 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:51.753334 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:51.753354 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:51.754619 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:51.754623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:51.755669 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:51.755682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:51.756527 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:51.756531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:51.756729 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:51.757421 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:51.757449 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:51.759247 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:51.761488 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:51.761495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:51.761515 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:51.761864 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:51.762201 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:51.762204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:51.762206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:51.762521 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:51.763084 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:51.763095 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:51.763449 (+ 354us) client_negotiation.cc:769] Sending connection context
0806 12:23:51.763661 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 12:23:51.764014 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":139,"threads_started":1}
I20260806 12:23:51.766680 6269 heartbeater.cc:347] Connected to a master server at 127.28.210.126:43531
I20260806 12:23:51.767145 6269 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:51.767798 6269 heartbeater.cc:511] Master 127.28.210.126:43531 requested a full tablet report, sending...
I20260806 12:23:51.770175 5969 ts_manager.cc:194] Registered new tserver with Master: c920f906d56c44ee825b7726cbfa5388 (127.28.210.65:32881)
I20260806 12:23:51.772063 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:41677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:23:51.784075 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:43531
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:51.911724 6276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:51.912189 6276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:51.912258 6276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:51.917634 6276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:51.917761 6276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:51.917834 6276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:23:51.923920 6276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:43531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:51.925232 6276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:51.926414 6276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:51.937944 6286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:51.937947 6290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:51.937947 6288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:51.938179 6276 server_base.cc:1034] running on GCE node
I20260806 12:23:51.938832 6276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:51.939643 6276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:51.940845 6276 hybrid_clock.cc:648] HybridClock initialized: now 1786019031940821 us; error 51 us; skew 500 ppm
Aug 06 12:23:51 dist-test-slave-lgr5 krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019031, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:51.944321 6276 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:23:51.945809 6276 webserver.cc:460] Webserver started at http://127.28.210.66:44249/ using document root <none> and password file <none>
I20260806 12:23:51.946650 6276 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:51.946790 6276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:51.947090 6276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:51.949755 6276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "cf69413e10bf46cfb3b0f08f7beecc5c"
format_stamp: "Formatted at 2026-08-06 12:23:51 on dist-test-slave-lgr5"
server_key: "8808ccfaeb36b40f38f58095bae103bf"
server_key_iv: "0cd1c371be2d881c0ff62030690699bc"
server_key_version: "encryptionkey@0"
I20260806 12:23:51.950466 6276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "cf69413e10bf46cfb3b0f08f7beecc5c"
format_stamp: "Formatted at 2026-08-06 12:23:51 on dist-test-slave-lgr5"
server_key: "8808ccfaeb36b40f38f58095bae103bf"
server_key_iv: "0cd1c371be2d881c0ff62030690699bc"
server_key_version: "encryptionkey@0"
I20260806 12:23:51.955184 6276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 12:23:51.959074 6301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:51.960917 6276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 12:23:51.961136 6276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "cf69413e10bf46cfb3b0f08f7beecc5c"
format_stamp: "Formatted at 2026-08-06 12:23:51 on dist-test-slave-lgr5"
server_key: "8808ccfaeb36b40f38f58095bae103bf"
server_key_iv: "0cd1c371be2d881c0ff62030690699bc"
server_key_version: "encryptionkey@0"
I20260806 12:23:51.961295 6276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:51.991333 6276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:51.995280 6276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:51.995503 6276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:51.996232 6276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:51.997277 6276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:51.997341 6276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:51.997392 6276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:51.997422 6276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:52.009891 6276 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:44095
I20260806 12:23:52.009917 6468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:44095 every 8 connection(s)
I20260806 12:23:52.011602 6276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:23:52.013741 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 6276
I20260806 12:23:52.013870 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:23:52.014228 29513 external_mini_cluster.cc:1428] Setting key a222e6d0c11c9e2512dfaabf90cb2995
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019031, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
I20260806 12:23:52.030369 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:52.014995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:53553 (local address 127.28.210.126:43531)
0806 12:23:52.015330 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:52.015337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:52.016868 (+ 1531us) server_negotiation.cc:407] Connection header received
0806 12:23:52.018562 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:52.018565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:52.018646 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:52.018730 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:52.020194 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.020765 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.022021 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.022268 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.022369 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:52.025449 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:52.025454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:52.025456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:52.025514 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:52.027287 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:52.028126 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:52.028131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:52.028133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:52.028240 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:52.028777 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:52.028780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:52.028782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:52.029051 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:52.029145 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:52.030005 (+ 860us) server_negotiation.cc:299] Negotiation successful
0806 12:23:52.030179 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260806 12:23:52.031489 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:52.015564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:43531 (local address 127.28.210.66:53553)
0806 12:23:52.016496 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:52.016608 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:52.018255 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:52.018976 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:52.018985 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:52.019428 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:52.019987 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:52.020015 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.020909 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.020912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:52.021870 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:52.021883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.022444 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.022449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:52.022645 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:52.023315 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:52.023335 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:52.025150 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:52.027542 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:52.027552 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:52.027579 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:52.027938 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:52.028384 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:52.028388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:52.028389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:52.028619 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:52.029192 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:52.029199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:52.029603 (+ 404us) client_negotiation.cc:769] Sending connection context
0806 12:23:52.029833 (+ 230us) client_negotiation.cc:240] Negotiation successful
0806 12:23:52.030221 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":199,"threads_started":1}
I20260806 12:23:52.033109 6470 heartbeater.cc:347] Connected to a master server at 127.28.210.126:43531
I20260806 12:23:52.033469 6470 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:52.034193 6470 heartbeater.cc:511] Master 127.28.210.126:43531 requested a full tablet report, sending...
I20260806 12:23:52.035800 5969 ts_manager.cc:194] Registered new tserver with Master: cf69413e10bf46cfb3b0f08f7beecc5c (127.28.210.66:44095)
I20260806 12:23:52.036545 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:53553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:23:52.056362 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:43531
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:52.193351 6477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:52.193703 6477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:52.193773 6477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:52.198720 6477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:52.198858 6477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:52.198971 6477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:23:52.205076 6477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:43531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:52.206377 6477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:52.207523 6477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:23:52.220770 6477 server_base.cc:1034] running on GCE node
W20260806 12:23:52.220698 6483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:52.220698 6482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:52.220698 6489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:52.221560 6477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:52.222430 6477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:52.223634 6477 hybrid_clock.cc:648] HybridClock initialized: now 1786019032223617 us; error 40 us; skew 500 ppm
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019032, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:52.227720 6477 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:23:52.229192 6477 webserver.cc:460] Webserver started at http://127.28.210.67:39115/ using document root <none> and password file <none>
I20260806 12:23:52.229904 6477 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:52.229985 6477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:52.230206 6477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:52.232630 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "74960d4e12b2498bb52bf2cab61bc480"
format_stamp: "Formatted at 2026-08-06 12:23:52 on dist-test-slave-lgr5"
server_key: "ddd87e134341fe6c0a6a913dfba15f3c"
server_key_iv: "ba79829d720282754b6a537b1e6dcb09"
server_key_version: "encryptionkey@0"
I20260806 12:23:52.233263 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "74960d4e12b2498bb52bf2cab61bc480"
format_stamp: "Formatted at 2026-08-06 12:23:52 on dist-test-slave-lgr5"
server_key: "ddd87e134341fe6c0a6a913dfba15f3c"
server_key_iv: "ba79829d720282754b6a537b1e6dcb09"
server_key_version: "encryptionkey@0"
I20260806 12:23:52.238332 6477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 12:23:52.241997 6500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:52.243762 6477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 12:23:52.243953 6477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "74960d4e12b2498bb52bf2cab61bc480"
format_stamp: "Formatted at 2026-08-06 12:23:52 on dist-test-slave-lgr5"
server_key: "ddd87e134341fe6c0a6a913dfba15f3c"
server_key_iv: "ba79829d720282754b6a537b1e6dcb09"
server_key_version: "encryptionkey@0"
I20260806 12:23:52.244150 6477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:52.261523 6477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:52.265722 6477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:52.265933 6477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:52.266675 6477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:52.267673 6477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:52.267731 6477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:52.267793 6477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:52.267819 6477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:52.279493 6477 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:45459
I20260806 12:23:52.279516 6695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:45459 every 8 connection(s)
I20260806 12:23:52.280908 6477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:23:52.286938 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 6477
I20260806 12:23:52.287112 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:23:52.287437 29513 external_mini_cluster.cc:1428] Setting key f7f25439696bd4462040bb17d18b7516
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019032, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:23:52.299075 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:52.283595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:59905 (local address 127.28.210.126:43531)
0806 12:23:52.283825 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:52.283831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:52.285070 (+ 1239us) server_negotiation.cc:407] Connection header received
0806 12:23:52.286826 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:52.286831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:52.286923 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:52.287043 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:52.288892 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.289487 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.290844 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.291088 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.291214 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:52.294337 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:52.294341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:52.294343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:52.294382 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:52.295938 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:52.296889 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:52.296895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:52.296896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:52.297100 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:52.297700 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:52.297706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:52.297708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:52.298002 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:52.298106 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:52.298780 (+ 674us) server_negotiation.cc:299] Negotiation successful
0806 12:23:52.298918 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 12:23:52.300048 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:52.284104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:43531 (local address 127.28.210.67:59905)
0806 12:23:52.284835 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:52.284901 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:52.286480 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:52.287358 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:52.287368 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:52.287884 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:52.288727 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:52.288749 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.289637 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.289640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:52.290687 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:52.290703 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.291333 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.291337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:52.291569 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:52.292269 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:52.292288 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:52.294044 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:52.296171 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:52.296179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:52.296200 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:52.296615 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:52.297244 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:52.297247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:52.297248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:52.297482 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:52.298122 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:52.298130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:52.298491 (+ 361us) client_negotiation.cc:769] Sending connection context
0806 12:23:52.298685 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 12:23:52.299019 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":191,"threads_started":1}
I20260806 12:23:52.301573 6696 heartbeater.cc:347] Connected to a master server at 127.28.210.126:43531
I20260806 12:23:52.301888 6696 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:52.302584 6696 heartbeater.cc:511] Master 127.28.210.126:43531 requested a full tablet report, sending...
I20260806 12:23:52.303843 5969 ts_manager.cc:194] Registered new tserver with Master: 74960d4e12b2498bb52bf2cab61bc480 (127.28.210.67:45459)
I20260806 12:23:52.304548 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:59905
I20260806 12:23:52.316331 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/test-admin.keytab.
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:52.337745 29513 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 12:23:52.341396 6706 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:23:52.339609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57130 (local address 127.28.210.65:32881)
0806 12:23:52.339954 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:52.339963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:52.339994 (+ 31us) server_negotiation.cc:407] Connection header received
0806 12:23:52.340140 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:52.340147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:52.340364 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:52.340521 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:52.341221 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57130: BlockingRecv error: recv got EOF from 127.0.0.1:57130 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":104,"threads_started":1}
W20260806 12:23:52.341634 29513 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/krb5cc' of type FILE
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:52.342904 29513 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:52.344007 29513 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.65@KRBTEST.COM
I20260806 12:23:52.357276 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:52.345161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.28.210.65:32881)
0806 12:23:52.345418 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:52.345423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:52.345475 (+ 52us) server_negotiation.cc:407] Connection header received
0806 12:23:52.345742 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:52.345746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:52.345813 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:52.345891 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:52.346574 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.347434 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.348708 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.349180 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.349278 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:52.352179 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:52.352196 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:52.352213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:52.352251 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:52.354735 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:52.355329 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:52.355335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:52.355336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:52.355432 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:52.355974 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:52.355976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:52.355977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:52.356394 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:52.356646 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:52.356943 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 12:23:52.357133 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260806 12:23:52.362931 6706 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:23:52.361645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57156 (local address 127.28.210.65:32881)
0806 12:23:52.361779 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:52.361784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:52.361969 (+ 185us) server_negotiation.cc:407] Connection header received
0806 12:23:52.362129 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:52.362132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:52.362188 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:52.362260 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:52.362806 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57156: BlockingRecv error: recv got EOF from 127.0.0.1:57156 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 12:23:52.363399 29513 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/krb5cc)
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:52.364610 29513 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:52.365648 29513 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.65@KRBTEST.COM
I20260806 12:23:52.376562 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:52.366804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.28.210.65:32881)
0806 12:23:52.366938 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:52.366942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:52.367054 (+ 112us) server_negotiation.cc:407] Connection header received
0806 12:23:52.367186 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:52.367189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:52.367238 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:52.367320 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:52.368002 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.368637 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.369652 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.370008 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.370115 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:52.372879 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:52.372883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:52.372886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:52.372923 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:52.374464 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:52.374954 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:52.374958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:52.374959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:52.375061 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:52.375330 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:52.375335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:52.375417 (+ 82us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:52.375670 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:52.375878 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:52.376203 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 12:23:52.376385 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 12:23:52.379967 6706 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:23:52.378777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.28.210.65:32881)
0806 12:23:52.378988 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:52.378992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:52.379101 (+ 109us) server_negotiation.cc:407] Connection header received
0806 12:23:52.379245 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:52.379248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:52.379294 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:52.379362 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:52.379845 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57172: BlockingRecv error: recv got EOF from 127.0.0.1:57172 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260806 12:23:52.380203 29513 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786018995816160-29513-0/krb5kdc/krb5cc' of type FILE
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:52.381282 29513 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:52.382172 29513 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[5894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.65@KRBTEST.COM
I20260806 12:23:52.392673 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:52.383667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.28.210.65:32881)
0806 12:23:52.383861 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:52.383866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:52.383949 (+ 83us) server_negotiation.cc:407] Connection header received
0806 12:23:52.384145 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:52.384149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:52.384203 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:52.384277 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:52.384914 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.385454 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.386284 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:52.386515 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:52.386613 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:52.389179 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:52.389184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:52.389186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:52.389221 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:52.390597 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:52.391078 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:52.391082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:52.391084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:52.391174 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:52.391533 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:52.391537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:52.391539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:52.391751 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:52.391913 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:52.392346 (+ 433us) server_negotiation.cc:299] Negotiation successful
0806 12:23:52.392487 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 12:23:52.394032 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 6066
I20260806 12:23:52.423588 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 6276
I20260806 12:23:52.468233 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 6477
I20260806 12:23:52.483560 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 5914
2026-08-06T12:23:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2651 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[6725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[6725](info): set up 2 sockets
Aug 06 12:23:52 dist-test-slave-lgr5 krb5kdc[6725](info): commencing operation
krb5kdc: starting...
W20260806 12:23:53.716744 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:23:53 dist-test-slave-lgr5 krb5kdc[6725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:23:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:23:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.126.keytab.
Entry for principal oryx/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.126.keytab.
I20260806 12:23:53.856333 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:37041
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.126.keytab
--principal=oryx/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:37041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:53.986956 6744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:53.987309 6744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:53.987385 6744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:53.992579 6744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:53.992697 6744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:53.992731 6744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:53.992761 6744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:53.992805 6744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:53.999293 6744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:37041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.126.keytab
--principal=oryx/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:37041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:54.000980 6744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:54.002686 6744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:54.013134 6752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:54.013206 6744 server_base.cc:1034] running on GCE node
W20260806 12:23:54.013134 6753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:54.013065 6758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:54.014412 6744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:54.015982 6744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:54.017244 6744 hybrid_clock.cc:648] HybridClock initialized: now 1786019034017226 us; error 57 us; skew 500 ppm
Aug 06 12:23:54 dist-test-slave-lgr5 krb5kdc[6725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019034, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:54.024881 6744 init.cc:377] Logged in from keytab as oryx/127.28.210.126@KRBTEST.COM (short username oryx)
I20260806 12:23:54.026089 6744 webserver.cc:460] Webserver started at http://127.28.210.126:33531/ using document root <none> and password file <none>
I20260806 12:23:54.026731 6744 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:54.026808 6744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:54.027043 6744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:54.029417 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "82a2d105a1274cba837b796cd94f697c"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "5f6d57e48f8395ab8d6498a26759b536"
server_key_iv: "62950f257ca7dd71b22a99526ed71879"
server_key_version: "encryptionkey@0"
I20260806 12:23:54.030061 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "82a2d105a1274cba837b796cd94f697c"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "5f6d57e48f8395ab8d6498a26759b536"
server_key_iv: "62950f257ca7dd71b22a99526ed71879"
server_key_version: "encryptionkey@0"
I20260806 12:23:54.034842 6744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 12:23:54.038041 6770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:54.039644 6744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:23:54.039849 6744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "82a2d105a1274cba837b796cd94f697c"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "5f6d57e48f8395ab8d6498a26759b536"
server_key_iv: "62950f257ca7dd71b22a99526ed71879"
server_key_version: "encryptionkey@0"
I20260806 12:23:54.040009 6744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:54.061781 6744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:54.065743 6744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:54.066000 6744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:54.074623 6744 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:37041
I20260806 12:23:54.074990 6842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:37041 every 8 connection(s)
I20260806 12:23:54.075891 6744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:23:54.076196 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 6744
I20260806 12:23:54.076368 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:23:54.076840 29513 external_mini_cluster.cc:1428] Setting key 75477dcea5a9bf81a74eb2884d739f1c
I20260806 12:23:54.080320 6844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 12:23:54 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:23:54.088631 6844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c: Bootstrap starting.
I20260806 12:23:54.091558 6844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:54.092548 6844 log.cc:824] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:54.094058 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:54.078620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.28.210.126:37041)
0806 12:23:54.079260 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:54.079280 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:54.079339 (+ 59us) server_negotiation.cc:407] Connection header received
0806 12:23:54.080350 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:54.080378 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:54.080850 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:54.081269 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:54.082361 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.083650 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.084873 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.085521 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.085742 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:54.088692 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:54.088699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:54.088717 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:54.088762 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:54.090626 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:54.091204 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:54.091211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:54.091218 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:54.091364 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:54.091664 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:54.091667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:54.091669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:54.092261 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:54.092476 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:54.092813 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 12:23:54.093180 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":202,"threads_started":1}
I20260806 12:23:54.095043 6844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c: No bootstrap required, opened a new log
I20260806 12:23:54.099222 6844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } }
I20260806 12:23:54.099540 6844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:54.099615 6844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a2d105a1274cba837b796cd94f697c, State: Initialized, Role: FOLLOWER
I20260806 12:23:54.100069 6844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } }
I20260806 12:23:54.100281 6844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:23:54.100354 6844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:23:54.100476 6844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:54.101598 6844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } }
I20260806 12:23:54.102021 6844 leader_election.cc:302] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82a2d105a1274cba837b796cd94f697c; no voters:
I20260806 12:23:54.102453 6844 leader_election.cc:288] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:23:54.102541 6850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:54.102787 6850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 1 LEADER]: Becoming Leader. State: Replica: 82a2d105a1274cba837b796cd94f697c, State: Running, Role: LEADER
I20260806 12:23:54.103169 6850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } }
I20260806 12:23:54.103991 6844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:54.105137 6853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82a2d105a1274cba837b796cd94f697c. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } }
I20260806 12:23:54.105289 6853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:54.105605 6852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } }
I20260806 12:23:54.105715 6852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: This master's current role is: LEADER
W20260806 12:23:54.109531 6868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:23:54.109632 6868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:23:54.109719 6861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:54.110491 6861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:54.119030 6861 catalog_manager.cc:1347] Generated new cluster ID: 8165e75ab35f4f5aadc5653b708b0362
I20260806 12:23:54.119154 6861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:54.145391 6861 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:23:54.146785 6861 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:54.157124 6861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c: Generated new TSK 0
I20260806 12:23:54.158026 6861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab.
Entry for principal oryx/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab.
I20260806 12:23:54.199407 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:37041
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab
--principal=oryx/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:54.326756 6876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:54.327133 6876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:54.327184 6876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:54.331790 6876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:54.331898 6876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:54.331964 6876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:23:54.339242 6876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab
--principal=oryx/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:37041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:54.340641 6876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:54.341789 6876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:54.352525 6891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:54.352526 6889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:54.352643 6876 server_base.cc:1034] running on GCE node
W20260806 12:23:54.352527 6887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:54.353320 6876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:54.354085 6876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:54.355266 6876 hybrid_clock.cc:648] HybridClock initialized: now 1786019034355245 us; error 38 us; skew 500 ppm
Aug 06 12:23:54 dist-test-slave-lgr5 krb5kdc[6725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019034, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:54.358338 6876 init.cc:377] Logged in from keytab as oryx/127.28.210.65@KRBTEST.COM (short username oryx)
I20260806 12:23:54.359369 6876 webserver.cc:460] Webserver started at http://127.28.210.65:44207/ using document root <none> and password file <none>
I20260806 12:23:54.359970 6876 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:54.360036 6876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:54.360266 6876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:54.362519 6876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "459f935f051e42d4938e20b9e58425b2"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "77332d99401f6e0e1bd942714c9dc1c6"
server_key_iv: "ce89de5ba50913462c1fcee0794ecc0d"
server_key_version: "encryptionkey@0"
I20260806 12:23:54.363134 6876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "459f935f051e42d4938e20b9e58425b2"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "77332d99401f6e0e1bd942714c9dc1c6"
server_key_iv: "ce89de5ba50913462c1fcee0794ecc0d"
server_key_version: "encryptionkey@0"
I20260806 12:23:54.369019 6876 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260806 12:23:54.372289 6900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:54.374039 6876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 12:23:54.374198 6876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "459f935f051e42d4938e20b9e58425b2"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "77332d99401f6e0e1bd942714c9dc1c6"
server_key_iv: "ce89de5ba50913462c1fcee0794ecc0d"
server_key_version: "encryptionkey@0"
I20260806 12:23:54.374334 6876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:54.390532 6876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:54.394301 6876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:54.394518 6876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:54.395226 6876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:54.396262 6876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:54.396319 6876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:54.396384 6876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:54.396411 6876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:54.406078 6876 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:37747
I20260806 12:23:54.406111 7083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:37747 every 8 connection(s)
I20260806 12:23:54.407122 6876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:23:54.408519 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 6876
I20260806 12:23:54.408644 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:23:54.408939 29513 external_mini_cluster.cc:1428] Setting key 5d1907b36a35442431f3685b66b7ebec
Aug 06 12:23:54 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019034, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:23:54.423647 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:54.409224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:42113 (local address 127.28.210.126:37041)
0806 12:23:54.409424 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:54.409429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:54.410510 (+ 1081us) server_negotiation.cc:407] Connection header received
0806 12:23:54.412036 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:54.412039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:54.412133 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:54.412212 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:54.413697 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.414333 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.415616 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.415879 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.415996 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:54.419508 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:54.419512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:54.419514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:54.419554 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:54.421118 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:54.421811 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:54.421816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:54.421818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:54.421925 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:54.422374 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:54.422377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:54.422378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:54.422600 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:54.422686 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:54.423336 (+ 650us) server_negotiation.cc:299] Negotiation successful
0806 12:23:54.423505 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.66@KRBTEST.COM" created.
I20260806 12:23:54.424875 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:54.409692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.65:42113)
0806 12:23:54.410335 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:54.410405 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:54.411774 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:54.412436 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:54.412444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:54.412948 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:54.413505 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:54.413522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.414483 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.414485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:54.415474 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:54.415486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.416125 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.416131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:54.416411 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:54.417137 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:54.417163 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:54.419190 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:54.421312 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:54.421319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:54.421338 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:54.421675 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:54.422035 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:54.422037 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:54.422039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:54.422224 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:54.422707 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:54.422714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:54.423055 (+ 341us) client_negotiation.cc:769] Sending connection context
0806 12:23:54.423264 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 12:23:54.423623 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":168,"threads_started":1}
I20260806 12:23:54.426551 7086 heartbeater.cc:347] Connected to a master server at 127.28.210.126:37041
I20260806 12:23:54.426857 7086 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:54.427568 7086 heartbeater.cc:511] Master 127.28.210.126:37041 requested a full tablet report, sending...
I20260806 12:23:54.429692 6796 ts_manager.cc:194] Registered new tserver with Master: 459f935f051e42d4938e20b9e58425b2 (127.28.210.65:37747)
I20260806 12:23:54.431012 6796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:42113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab.
Entry for principal oryx/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab.
I20260806 12:23:54.444408 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:37041
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab
--principal=oryx/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:54.582417 7091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:54.582866 7091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:54.582944 7091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:54.587414 7091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:54.587512 7091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:54.587605 7091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:23:54.593866 7091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab
--principal=oryx/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:37041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:54.595173 7091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:54.596415 7091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:54.607865 7099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:54.607934 7091 server_base.cc:1034] running on GCE node
W20260806 12:23:54.607865 7098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:54.607884 7103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:54.608703 7091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:54.609508 7091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:54.610721 7091 hybrid_clock.cc:648] HybridClock initialized: now 1786019034610680 us; error 66 us; skew 500 ppm
Aug 06 12:23:54 dist-test-slave-lgr5 krb5kdc[6725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019034, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:54.614048 7091 init.cc:377] Logged in from keytab as oryx/127.28.210.66@KRBTEST.COM (short username oryx)
I20260806 12:23:54.615298 7091 webserver.cc:460] Webserver started at http://127.28.210.66:40255/ using document root <none> and password file <none>
I20260806 12:23:54.615990 7091 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:54.616066 7091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:54.616322 7091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:54.618870 7091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "55b89e5769ac4aeabaa157a85d382d85"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "15995113bde3c4b3b84548910822f6bc"
server_key_iv: "ef3ce55b7a928023efc5b35826cf1f63"
server_key_version: "encryptionkey@0"
I20260806 12:23:54.619592 7091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "55b89e5769ac4aeabaa157a85d382d85"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "15995113bde3c4b3b84548910822f6bc"
server_key_iv: "ef3ce55b7a928023efc5b35826cf1f63"
server_key_version: "encryptionkey@0"
I20260806 12:23:54.624706 7091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260806 12:23:54.627847 7112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:54.629961 7091 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:23:54.630169 7091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "55b89e5769ac4aeabaa157a85d382d85"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "15995113bde3c4b3b84548910822f6bc"
server_key_iv: "ef3ce55b7a928023efc5b35826cf1f63"
server_key_version: "encryptionkey@0"
I20260806 12:23:54.630324 7091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:54.641105 7091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:54.644846 7091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:54.645030 7091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:54.645642 7091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:54.646559 7091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:54.646608 7091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:54.646651 7091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:54.646673 7091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:54.656031 7091 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:41305
I20260806 12:23:54.656174 7277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:41305 every 8 connection(s)
I20260806 12:23:54.657042 7091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:23:54.664312 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 7091
I20260806 12:23:54.664433 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:23:54.664695 29513 external_mini_cluster.cc:1428] Setting key 3fb37b3997c9ee99926f62bb2208dc96
Aug 06 12:23:54 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019034, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:23:54.671932 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:54.659064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:39581 (local address 127.28.210.126:37041)
0806 12:23:54.659205 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:54.659209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:54.660059 (+ 850us) server_negotiation.cc:407] Connection header received
0806 12:23:54.661512 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:54.661516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:54.661567 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:54.661643 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:54.662887 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.663444 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.664809 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.665050 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.665146 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:54.668176 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:54.668179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:54.668182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:54.668209 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:54.669697 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:54.670268 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:54.670271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:54.670272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:54.670354 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:54.670721 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:54.670725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:54.670726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:54.670928 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:54.671020 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:54.671615 (+ 595us) server_negotiation.cc:299] Negotiation successful
0806 12:23:54.671755 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 12:23:54.672900 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:54.659333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.66:39581)
0806 12:23:54.659859 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:54.659906 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:54.661286 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:54.661804 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:54.661812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:54.662211 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:54.662714 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:54.662729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.663617 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.663620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:54.664686 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:54.664696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.665210 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.665214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:54.665414 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:54.666154 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:54.666178 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:54.667952 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:54.669844 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:54.669851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:54.669866 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:54.670162 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:54.670459 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:54.670463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:54.670465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:54.670625 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:54.671041 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:54.671049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:54.671354 (+ 305us) client_negotiation.cc:769] Sending connection context
0806 12:23:54.671545 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 12:23:54.671921 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":124,"threads_started":1}
I20260806 12:23:54.674228 7278 heartbeater.cc:347] Connected to a master server at 127.28.210.126:37041
I20260806 12:23:54.674552 7278 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:54.675298 7278 heartbeater.cc:511] Master 127.28.210.126:37041 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.28.210.67@KRBTEST.COM; defaulting to no policy
I20260806 12:23:54.676899 6796 ts_manager.cc:194] Registered new tserver with Master: 55b89e5769ac4aeabaa157a85d382d85 (127.28.210.66:41305)
I20260806 12:23:54.677572 6796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:39581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab.
Entry for principal oryx/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab.
I20260806 12:23:54.701272 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:37041
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab
--principal=oryx/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:54.834744 7289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:54.835111 7289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:54.835170 7289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:54.839834 7289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:54.839960 7289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:54.840034 7289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:23:54.847272 7289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab
--principal=oryx/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:37041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:54.849123 7289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:54.850376 7289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:54.861204 7295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:54.861263 7300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:54.861263 7298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:54.861510 7289 server_base.cc:1034] running on GCE node
I20260806 12:23:54.862040 7289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:54.862884 7289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:54.864127 7289 hybrid_clock.cc:648] HybridClock initialized: now 1786019034864058 us; error 45 us; skew 500 ppm
Aug 06 12:23:54 dist-test-slave-lgr5 krb5kdc[6725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019034, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:54.867657 7289 init.cc:377] Logged in from keytab as oryx/127.28.210.67@KRBTEST.COM (short username oryx)
I20260806 12:23:54.868897 7289 webserver.cc:460] Webserver started at http://127.28.210.67:46563/ using document root <none> and password file <none>
I20260806 12:23:54.869534 7289 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:54.869625 7289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:54.869903 7289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:54.872344 7289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "2e87c17242b84502bb0db4ad9d7cf416"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "1df665410c58a0b8fa76335ba61b71f6"
server_key_iv: "6ccc2b7417c1ca8b1ab6b7697e4b097e"
server_key_version: "encryptionkey@0"
I20260806 12:23:54.872989 7289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "2e87c17242b84502bb0db4ad9d7cf416"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "1df665410c58a0b8fa76335ba61b71f6"
server_key_iv: "6ccc2b7417c1ca8b1ab6b7697e4b097e"
server_key_version: "encryptionkey@0"
I20260806 12:23:54.877295 7289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:23:54.880124 7315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:54.881520 7289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:54.881690 7289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "2e87c17242b84502bb0db4ad9d7cf416"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "1df665410c58a0b8fa76335ba61b71f6"
server_key_iv: "6ccc2b7417c1ca8b1ab6b7697e4b097e"
server_key_version: "encryptionkey@0"
I20260806 12:23:54.881810 7289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:54.899269 7289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:54.903123 7289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:54.903362 7289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:54.904345 7289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:54.905535 7289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:54.905599 7289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:54.905664 7289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:54.905696 7289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:54.919886 7289 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:33181
I20260806 12:23:54.919924 7502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:33181 every 8 connection(s)
I20260806 12:23:54.921137 7289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:23:54.922266 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 7289
I20260806 12:23:54.922397 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:23:54.922753 29513 external_mini_cluster.cc:1428] Setting key 37dc4f6b26728a92d05c19718c315bdc
Aug 06 12:23:54 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019034, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:23:54.939666 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:54.923695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:43329 (local address 127.28.210.126:37041)
0806 12:23:54.923851 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:54.923855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:54.925037 (+ 1182us) server_negotiation.cc:407] Connection header received
0806 12:23:54.926869 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:54.926873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:54.926943 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:54.927030 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:54.928553 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.929126 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.930511 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.930754 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.930875 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:54.934560 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:54.934564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:54.934566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:54.934602 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:54.936342 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:54.937197 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:54.937201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:54.937202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:54.937299 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:54.937953 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:54.937959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:54.937961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:54.938431 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:54.938560 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:54.939353 (+ 793us) server_negotiation.cc:299] Negotiation successful
0806 12:23:54.939500 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 12:23:54.940811 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:54.924198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.67:43329)
0806 12:23:54.924867 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:54.924921 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:54.926591 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:54.927255 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:54.927263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:54.927711 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:54.928348 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:54.928369 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.929303 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.929307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:54.930339 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:54.930353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:54.930959 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:54.930964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:54.931187 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:54.932041 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:54.932063 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:54.934243 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:54.936605 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:54.936615 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:54.936639 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:54.937038 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:54.937397 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:54.937401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:54.937404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:54.937699 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:54.938610 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:54.938618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:54.939027 (+ 409us) client_negotiation.cc:769] Sending connection context
0806 12:23:54.939266 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 12:23:54.939652 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":180,"threads_started":1}
I20260806 12:23:54.943029 7504 heartbeater.cc:347] Connected to a master server at 127.28.210.126:37041
I20260806 12:23:54.943370 7504 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:54.944026 7504 heartbeater.cc:511] Master 127.28.210.126:37041 requested a full tablet report, sending...
I20260806 12:23:54.945768 6796 ts_manager.cc:194] Registered new tserver with Master: 2e87c17242b84502bb0db4ad9d7cf416 (127.28.210.67:33181)
I20260806 12:23:54.946494 6796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:43329
I20260806 12:23:54.952380 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.210.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.125.keytab.
Entry for principal oryx/127.28.210.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.125.keytab.
I20260806 12:23:54.991884 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.125:46317
--webserver_interface=127.28.210.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.125.keytab
--principal=oryx/127.28.210.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:37041,127.28.210.125:46317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:23:55.131963 7509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:55.132450 7509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:55.132535 7509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:55.137300 7509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:55.137409 7509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:55.137444 7509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:55.137473 7509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:55.137511 7509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:55.144043 7509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:37041,127.28.210.125:46317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.125.keytab
--principal=oryx/127.28.210.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.125:46317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.210.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
W20260806 12:23:55.144429 7509 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.210.126:37041,127.28.210.125:46317'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 12:23:55.145561 7509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:55.146751 7509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:55.157074 7516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:55.157063 7519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:55.157132 7509 server_base.cc:1034] running on GCE node
W20260806 12:23:55.157056 7515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:55.158290 7509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:55.159765 7509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:55.161022 7509 hybrid_clock.cc:648] HybridClock initialized: now 1786019035160982 us; error 62 us; skew 500 ppm
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:55.165102 7509 init.cc:377] Logged in from keytab as oryx/127.28.210.125@KRBTEST.COM (short username oryx)
I20260806 12:23:55.166332 7509 webserver.cc:460] Webserver started at http://127.28.210.125:43707/ using document root <none> and password file <none>
I20260806 12:23:55.167030 7509 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:55.167104 7509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:55.167356 7509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:55.169883 7509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data/instance:
uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a"
format_stamp: "Formatted at 2026-08-06 12:23:55 on dist-test-slave-lgr5"
server_key: "5d52231896196d0005b2f29cba293022"
server_key_iv: "5f6b38f528771f188db004ed31740658"
server_key_version: "encryptionkey@0"
I20260806 12:23:55.170567 7509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/wal/instance:
uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a"
format_stamp: "Formatted at 2026-08-06 12:23:55 on dist-test-slave-lgr5"
server_key: "5d52231896196d0005b2f29cba293022"
server_key_iv: "5f6b38f528771f188db004ed31740658"
server_key_version: "encryptionkey@0"
I20260806 12:23:55.175382 7509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 12:23:55.178764 7532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:55.180617 7509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:55.180810 7509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/wal
uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a"
format_stamp: "Formatted at 2026-08-06 12:23:55 on dist-test-slave-lgr5"
server_key: "5d52231896196d0005b2f29cba293022"
server_key_iv: "5f6b38f528771f188db004ed31740658"
server_key_version: "encryptionkey@0"
I20260806 12:23:55.180962 7509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:55.204414 7509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:55.208472 7509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:55.208729 7509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:55.217358 7509 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.125:46317
I20260806 12:23:55.217705 7613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.125:46317 every 8 connection(s)
I20260806 12:23:55.218519 7509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
I20260806 12:23:55.221550 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 7509
I20260806 12:23:55.221899 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:55.227862 7614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.210.125@KRBTEST.COM
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.125@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:23:55.239276 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.223420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43492 (local address 127.28.210.125:46317)
0806 12:23:55.223936 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.223960 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.224010 (+ 50us) server_negotiation.cc:407] Connection header received
0806 12:23:55.225070 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.225099 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.225616 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.226081 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.226839 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.228510 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.229445 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.230045 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.230228 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.233557 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.233563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.233575 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.233609 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.235873 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.236428 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.236435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.236442 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.236592 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.236894 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.236897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.236898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.237355 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.237587 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.237905 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.238227 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":144,"threads_started":1}
I20260806 12:23:55.240381 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.230222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.28.210.126:37041)
0806 12:23:55.230388 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.230393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.230818 (+ 425us) server_negotiation.cc:407] Connection header received
0806 12:23:55.231072 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.231075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.231124 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.231185 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.232181 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.232706 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.233972 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.234287 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.234372 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.236957 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.236961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.236967 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.236997 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.238482 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.238948 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.238951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.238953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.239037 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.239321 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.239324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.239325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.239598 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.239721 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.240059 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.240220 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 12:23:55.240404 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.230129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:37786)
0806 12:23:55.230682 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.230728 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.230936 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.231295 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.231303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.231792 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.231991 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.232001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.232846 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.232849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.233798 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.233812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.234427 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.234431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.234568 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.235049 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.235065 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.236779 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.238593 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.238603 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.238611 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.238838 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.239128 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.239130 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.239132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.239218 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.239719 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.239727 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.239849 (+ 122us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.239981 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.240225 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":160,"threads_started":1}
I20260806 12:23:55.241808 7614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.125@KRBTEST.COM for oryx/127.28.210.125@KRBTEST.COM
I20260806 12:23:55.250808 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.242442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.28.210.125:46317)
0806 12:23:55.242534 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.242538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.242550 (+ 12us) server_negotiation.cc:407] Connection header received
0806 12:23:55.242618 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.242621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.242657 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.242718 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.243114 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.243642 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.245250 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.245521 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.245612 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.247767 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.247772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.247775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.247802 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.249172 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.249589 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.249592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.249594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.249680 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.249961 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.249964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.249966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.250165 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.250251 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.250522 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.250654 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":38}
I20260806 12:23:55.250815 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.242324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:46317 (local address 127.0.0.1:43506)
0806 12:23:55.242452 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.242469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.242549 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.242740 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.242743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.242908 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.243010 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.243015 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.243738 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.243741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.245133 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.245147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.245626 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.245630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.245740 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.246230 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.246241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.247612 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.249298 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.249301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.249303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.249498 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.249763 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.249766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.249768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.249876 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.250256 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.250259 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.250350 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.250441 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.250605 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":20}
I20260806 12:23:55.254448 7614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: Bootstrap starting.
I20260806 12:23:55.257609 7614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:55.258571 7614 log.cc:824] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:55.261099 7614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: No bootstrap required, opened a new log
I20260806 12:23:55.264629 7614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } }
I20260806 12:23:55.264915 7614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:55.264958 7614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7a6f38f9fc40f187bf1ef7334aca6a, State: Initialized, Role: FOLLOWER
I20260806 12:23:55.265491 7614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } }
I20260806 12:23:55.266083 7626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } } }
I20260806 12:23:55.266237 7626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:23:55.267037 7614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: configured and running, proceeding with master startup.
W20260806 12:23:55.272593 7645 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:23:55.272719 7645 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:23:55.275827 7509 master_runner.cc:428] Detected that this master 3c7a6f38f9fc40f187bf1ef7334aca6a is joining an existing cluster
I20260806 12:23:55.275940 7509 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.210.125:46317
I20260806 12:23:55.288888 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.279870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.28.210.125:46317)
0806 12:23:55.280015 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.280021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.280336 (+ 315us) server_negotiation.cc:407] Connection header received
0806 12:23:55.280393 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.280396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.280451 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.280560 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.281476 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.282291 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.283453 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.283669 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.283765 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.285267 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.285270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.285273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.285295 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.287082 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.287536 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.287540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.287542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.287640 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.287927 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.287931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.287933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.288201 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.288298 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.288546 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.288748 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":49}
I20260806 12:23:55.288901 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.279320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:46317 (local address 127.0.0.1:43514)
0806 12:23:55.280129 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.280146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.280305 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.280602 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.280605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.281026 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.281325 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.281332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.282535 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.282539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.283350 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.283358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.283804 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.283807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.283909 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.284557 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.284571 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.285121 (+ 550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.287189 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.287193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.287195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.287408 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.287741 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.287744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.287746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.287842 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.288297 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.288300 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.288393 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.288486 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.288674 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":58,"spinlock_wait_cycles":20352,"thread_start_us":185,"threads_started":1}
I20260806 12:23:55.289285 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.279541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.28.210.126:37041)
0806 12:23:55.279906 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.279912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.280285 (+ 373us) server_negotiation.cc:407] Connection header received
0806 12:23:55.280452 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.280456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.280511 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.280591 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.281500 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.282055 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.283331 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.283723 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.283874 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.285470 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.285474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.285477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.285511 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.287348 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.287818 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.287822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.287824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.287919 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.288287 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.288290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.288292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.288529 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.288608 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.288981 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.289114 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260806 12:23:55.289827 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.279320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:37788)
0806 12:23:55.280145 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.280157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.280305 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.280732 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.280734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.281025 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.281322 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.281332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.282184 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.282188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.283155 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.283168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.284038 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.284042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.284210 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.284672 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.284688 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.285280 (+ 592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.287467 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.287470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.287472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.287675 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.288032 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.288036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.288038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.288172 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.288683 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.288685 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.288780 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.289451 (+ 671us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.289630 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":79,"spinlock_wait_cycles":11520,"thread_start_us":141,"threads_started":1}
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019034, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.126@KRBTEST.COM for oryx/127.28.210.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.28.210.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.124@KRBTEST.COM" created.
I20260806 12:23:55.307834 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.296434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.28.210.125:46317)
0806 12:23:55.296567 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.296572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.296586 (+ 14us) server_negotiation.cc:407] Connection header received
0806 12:23:55.296662 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.296665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.296720 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.296807 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.297775 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.298473 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.299826 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.300207 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.300329 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.303317 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.303321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.303323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.303353 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.305223 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.306069 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.306074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.306075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.306172 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.306536 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.306540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.306541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.306884 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.306997 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.307476 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.307648 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 12:23:55.307854 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.295537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:46317 (local address 127.0.0.1:43520)
0806 12:23:55.295998 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.296057 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.296240 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.296891 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.296898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.297428 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.297617 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.297628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.298597 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.298601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.299646 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.299662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.300344 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.300348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.300483 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.301162 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.301181 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.303119 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.305344 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.305362 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.305372 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.305932 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.306267 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.306270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.306273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.306389 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.307021 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.307030 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.307185 (+ 155us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.307381 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.307679 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":121,"threads_started":1}
I20260806 12:23:55.310359 6796 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.210.125:46317: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a"
member_type: NON_VOTER
last_known_addr {
host: "127.28.210.125"
port: 46317
}
attrs {
promote: true
}
}
dest_uuid: "82a2d105a1274cba837b796cd94f697c"
cas_config_opid_index: -1
I20260806 12:23:55.311318 6796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true } }
I20260806 12:23:55.313514 7666 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3c7a6f38f9fc40f187bf1ef7334aca6a (127.28.210.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true } } }
I20260806 12:23:55.314323 7577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:55.315586 7666 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.210.125:46317
I20260806 12:23:55.315945 7577 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 FOLLOWER]: Refusing update from remote peer 82a2d105a1274cba837b796cd94f697c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 12:23:55.316200 7666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true } } }
I20260806 12:23:55.316362 7666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:55.316524 7509 master.cc:537] Master@127.28.210.125:46317 shutting down...
I20260806 12:23:55.318226 7666 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:55.322393 7509 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 12:23:55.322659 6771 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260806 12:23:55.322736 7509 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:55.322845 7509 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: stopping tablet replica
W20260806 12:23:55.325368 6771 consensus_peers.cc:487] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c -> Peer 3c7a6f38f9fc40f187bf1ef7334aca6a (127.28.210.125:46317): Couldn't send request to peer 3c7a6f38f9fc40f187bf1ef7334aca6a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.124.keytab.
Entry for principal oryx/127.28.210.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.124.keytab.
I20260806 12:23:55.331744 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.124:39995
--webserver_interface=127.28.210.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.124.keytab
--principal=oryx/127.28.210.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:37041,127.28.210.125:46317,127.28.210.124:39995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:23:55.339362 7509 master.cc:552] Master@127.28.210.125:46317 shutdown complete.
I20260806 12:23:55.339434 7509 master_runner.cc:305] Clearing existing system tablet
I20260806 12:23:55.339694 7509 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:55.339752 7509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:55.341014 7509 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260806 12:23:55.341879 7674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:55.342126 7509 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:55.342262 7509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 12:23:55.342303 7509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/wal
uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a"
format_stamp: "Formatted at 2026-08-06 12:23:55 on dist-test-slave-lgr5"
server_key: "5d52231896196d0005b2f29cba293022"
server_key_iv: "5f6b38f528771f188db004ed31740658"
server_key_version: "encryptionkey@0"
I20260806 12:23:55.344547 7509 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:55.346993 7509 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 12:23:55.347105 7509 log.cc:1196] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 12:23:55.347386 7509 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: Deleting consensus metadata
I20260806 12:23:55.347656 7509 master_runner.cc:315] Copying system tablet from 127.28.210.126:37041
I20260806 12:23:55.349156 7509 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: tablet copy: Beginning tablet copy session from remote peer at address 127.28.210.126:37041
I20260806 12:23:55.357723 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.349936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.28.210.126:37041)
0806 12:23:55.350076 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.350081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.350322 (+ 241us) server_negotiation.cc:407] Connection header received
0806 12:23:55.350495 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.350498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.350553 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.350606 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.351315 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.351850 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.353033 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.353256 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.353348 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.354568 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.354571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.354573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.354601 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.355984 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.356492 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.356495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.356496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.356574 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.356836 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.356839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.356841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.357048 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.357142 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.357413 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.357546 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:23:55.358093 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.349826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:37796)
0806 12:23:55.350205 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.350218 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.350363 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.350814 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.350818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.351037 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.351162 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.351168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.351960 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.351971 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.352891 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.352904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.353366 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.353369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.353461 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.354031 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.354043 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.354394 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.356182 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.356185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.356187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.356390 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.356662 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.356665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.356668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.356752 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.357144 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.357147 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.357251 (+ 104us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.357342 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.357605 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":114,"threads_started":1}
I20260806 12:23:55.358883 6823 tablet_copy_service.cc:140] P 82a2d105a1274cba837b796cd94f697c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3c7a6f38f9fc40f187bf1ef7334aca6a ({username='oryx', principal='oryx/127.28.210.125@KRBTEST.COM'} at 127.0.0.1:37796)
I20260806 12:23:55.359140 6823 tablet_copy_service.cc:161] P 82a2d105a1274cba837b796cd94f697c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3c7a6f38f9fc40f187bf1ef7334aca6a at {username='oryx', principal='oryx/127.28.210.125@KRBTEST.COM'} at 127.0.0.1:37796: session id = 3c7a6f38f9fc40f187bf1ef7334aca6a-00000000000000000000000000000000
I20260806 12:23:55.361389 6823 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 12:23:55.363783 7509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 12:23:55.366659 7509 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: tablet copy: Starting download of 0 data blocks...
I20260806 12:23:55.366909 7509 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: tablet copy: Starting download of 1 WAL segments...
I20260806 12:23:55.369446 7509 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 12:23:55.371851 6823 tablet_copy_service.cc:342] P 82a2d105a1274cba837b796cd94f697c: Request end of tablet copy session 3c7a6f38f9fc40f187bf1ef7334aca6a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.210.125@KRBTEST.COM'} at 127.0.0.1:37796
I20260806 12:23:55.372020 6823 tablet_copy_service.cc:433] P 82a2d105a1274cba837b796cd94f697c: ending tablet copy session 3c7a6f38f9fc40f187bf1ef7334aca6a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3c7a6f38f9fc40f187bf1ef7334aca6a
I20260806 12:23:55.381316 7509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:55.383322 7681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:55.385144 7682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:55.385775 7509 server_base.cc:1034] running on GCE node
W20260806 12:23:55.388288 7684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:55.388581 7509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:55.388960 7509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:55.390116 7509 hybrid_clock.cc:648] HybridClock initialized: now 1786019035390095 us; error 42 us; skew 500 ppm
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:55.391824 7509 init.cc:377] Logged in from keytab as oryx/127.28.210.125@KRBTEST.COM (short username oryx)
I20260806 12:23:55.392403 7509 webserver.cc:460] Webserver started at http://127.28.210.125:38167/ using document root <none> and password file <none>
I20260806 12:23:55.392617 7509 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:55.392663 7509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:55.394100 7509 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 12:23:55.395185 7696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:55.395467 7509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 12:23:55.395578 7509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/wal
uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a"
format_stamp: "Formatted at 2026-08-06 12:23:55 on dist-test-slave-lgr5"
server_key: "5d52231896196d0005b2f29cba293022"
server_key_iv: "5f6b38f528771f188db004ed31740658"
server_key_version: "encryptionkey@0"
I20260806 12:23:55.395704 7509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:55.415477 7509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:55.418247 7509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:55.424543 7509 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.125:46317
I20260806 12:23:55.425328 7509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
I20260806 12:23:55.424568 7766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.125:46317 every 8 connection(s)
I20260806 12:23:55.427532 7767 sys_catalog.cc:263] Verifying existing consensus state
I20260806 12:23:55.428709 7767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: Bootstrap starting.
I20260806 12:23:55.434100 7086 heartbeater.cc:503] Master 127.28.210.126:37041 was elected leader, sending a full tablet report...
I20260806 12:23:55.440927 7767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 12:23:55.441279 7767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: Bootstrap complete.
I20260806 12:23:55.441977 7767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true } }
I20260806 12:23:55.442157 7767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c7a6f38f9fc40f187bf1ef7334aca6a, State: Initialized, Role: LEARNER
I20260806 12:23:55.442274 7767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true } }
I20260806 12:23:55.442642 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true } } }
I20260806 12:23:55.442786 7767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:55.442828 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: This master's current role is: LEARNER
I20260806 12:23:55.447173 7789 catalog_manager.cc:1259] Loaded cluster ID: 8165e75ab35f4f5aadc5653b708b0362
I20260806 12:23:55.447243 7789 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: loading cluster ID for follower catalog manager: success
I20260806 12:23:55.449509 7789 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: acquiring CA information for follower catalog manager: success
I20260806 12:23:55.451017 7789 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 12:23:55.481948 7672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:55.482270 7672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:55.482345 7672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:55.486883 7672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:23:55.486971 7672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:55.486999 7672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:55.487023 7672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:23:55.487049 7672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:23:55.493127 7672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:37041,127.28.210.125:46317,127.28.210.124:39995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.124.keytab
--principal=oryx/127.28.210.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.124:39995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.210.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:55.494392 7672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:55.495596 7672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:55.504968 7799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:55.505004 7672 server_base.cc:1034] running on GCE node
W20260806 12:23:55.504949 7803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:55.504949 7800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:55.505972 7672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:55.507246 7672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:55.508430 7672 hybrid_clock.cc:648] HybridClock initialized: now 1786019035508408 us; error 35 us; skew 500 ppm
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:55.511497 7672 init.cc:377] Logged in from keytab as oryx/127.28.210.124@KRBTEST.COM (short username oryx)
I20260806 12:23:55.512566 7672 webserver.cc:460] Webserver started at http://127.28.210.124:38715/ using document root <none> and password file <none>
I20260806 12:23:55.513252 7672 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:55.513319 7672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:55.513509 7672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:23:55.515892 7672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data/instance:
uuid: "c0c3343c36004df38a7b11f49bf19adc"
format_stamp: "Formatted at 2026-08-06 12:23:55 on dist-test-slave-lgr5"
server_key: "002d080e7562e7a207c65160aa5462c9"
server_key_iv: "472e10ed7965ac1bfe139680fd0be7b1"
server_key_version: "encryptionkey@0"
I20260806 12:23:55.516628 7672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/wal/instance:
uuid: "c0c3343c36004df38a7b11f49bf19adc"
format_stamp: "Formatted at 2026-08-06 12:23:55 on dist-test-slave-lgr5"
server_key: "002d080e7562e7a207c65160aa5462c9"
server_key_iv: "472e10ed7965ac1bfe139680fd0be7b1"
server_key_version: "encryptionkey@0"
I20260806 12:23:55.520991 7672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:23:55.523679 7812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:55.525094 7672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 12:23:55.525238 7672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/wal
uuid: "c0c3343c36004df38a7b11f49bf19adc"
format_stamp: "Formatted at 2026-08-06 12:23:55 on dist-test-slave-lgr5"
server_key: "002d080e7562e7a207c65160aa5462c9"
server_key_iv: "472e10ed7965ac1bfe139680fd0be7b1"
server_key_version: "encryptionkey@0"
I20260806 12:23:55.525358 7672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:55.539534 7672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:55.542994 7672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:55.543215 7672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:55.555783 7672 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.124:39995
I20260806 12:23:55.556221 7886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.124:39995 every 8 connection(s)
I20260806 12:23:55.557322 7672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
I20260806 12:23:55.561206 7888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:55.561219 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 7672
I20260806 12:23:55.568387 7888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.210.124@KRBTEST.COM
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.124@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:23:55.580847 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.563173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45220 (local address 127.28.210.124:39995)
0806 12:23:55.563832 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.563866 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.563920 (+ 54us) server_negotiation.cc:407] Connection header received
0806 12:23:55.565005 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.565037 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.565600 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.566093 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.566862 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.568673 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.569736 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.570458 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.570636 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.574419 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.574428 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.574455 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.574498 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.576917 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.577568 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.577576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.577584 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.577712 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.578138 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.578141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.578143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.578757 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.579105 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.579341 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.579683 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":198,"threads_started":1}
I20260806 12:23:55.582767 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.571106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.28.210.126:37041)
0806 12:23:55.571252 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.571257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.571482 (+ 225us) server_negotiation.cc:407] Connection header received
0806 12:23:55.571628 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.571632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.571682 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.571750 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.572798 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.573578 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.575200 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.575573 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.575678 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.578952 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.578957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.578959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.578998 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.580632 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.581227 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.581231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.581233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.581336 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.581728 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.581731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.581733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.581978 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.582084 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.582529 (+ 445us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.582646 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 12:23:55.582928 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.570752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:37804)
0806 12:23:55.571333 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.571387 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.571534 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.571824 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.571831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.572382 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.572642 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.572652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.573820 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.573829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.574990 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.575005 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.575910 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.575914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.576057 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.576878 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.576934 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.578769 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.580762 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.580771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.580779 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.581076 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.581507 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.581511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.581512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.581622 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.582104 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.582111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.582299 (+ 188us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.582482 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.582706 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"spinlock_wait_cycles":768,"thread_start_us":201,"threads_started":1}
I20260806 12:23:55.584262 7888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.124@KRBTEST.COM for oryx/127.28.210.125@KRBTEST.COM
I20260806 12:23:55.592880 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.584802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:46317 (local address 127.0.0.1:43524)
0806 12:23:55.584924 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.584940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.585012 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.585390 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.585393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.585541 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.585697 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.585706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.586610 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.586613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.587196 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.587206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.587713 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.587716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.587821 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.588399 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.588413 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.589785 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.591467 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.591470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.591471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.591675 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.591924 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.591926 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.591927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.592003 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.592408 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.592410 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.592507 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.592594 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.592720 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 12:23:55.592916 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.584893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.28.210.125:46317)
0806 12:23:55.585185 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.585189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.585202 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:23:55.585243 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.585245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.585285 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.585368 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.585813 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.586505 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.587344 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.587590 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.587693 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.589986 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.589990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.589992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.590022 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.591340 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.591763 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.591765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.591767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.591849 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.592070 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.592072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.592073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.592305 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.592413 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.592658 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.592776 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":118,"threads_started":1}
I20260806 12:23:55.593943 7888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.124@KRBTEST.COM for oryx/127.28.210.124@KRBTEST.COM
I20260806 12:23:55.602944 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.594550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:39995 (local address 127.0.0.1:45232)
0806 12:23:55.594714 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.594733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.594812 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.595058 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.595061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.595227 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.595298 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.595304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.596883 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.596887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.597395 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.597404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.597847 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.597850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.597919 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.598407 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.598422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.599742 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.601495 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.601498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.601500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.601689 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.601942 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.601945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.601947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.602023 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.602386 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.602389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.602478 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.602566 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.602730 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 12:23:55.602944 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.594764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45232 (local address 127.28.210.124:39995)
0806 12:23:55.594877 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.594881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.594894 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:23:55.594925 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.594927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.594959 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.595035 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.595406 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.596738 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.597509 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.597733 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.597825 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.599902 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.599905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.599907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.599930 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.601375 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.601780 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.601784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.601786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.601858 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.602101 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.602104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.602106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.602295 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.602392 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.602622 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.602787 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":50}
I20260806 12:23:55.606825 7888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: Bootstrap starting.
I20260806 12:23:55.610026 7888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:55.611330 7888 log.cc:824] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:55.614456 7888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: No bootstrap required, opened a new log
I20260806 12:23:55.619050 7888 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } }
I20260806 12:23:55.619377 7888 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:55.619428 7888 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c3343c36004df38a7b11f49bf19adc, State: Initialized, Role: FOLLOWER
I20260806 12:23:55.619910 7888 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } }
I20260806 12:23:55.620555 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } } }
I20260806 12:23:55.620700 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:23:55.621433 7888 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: configured and running, proceeding with master startup.
W20260806 12:23:55.628157 7920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:23:55.628297 7920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:23:55.631657 7672 master_runner.cc:428] Detected that this master c0c3343c36004df38a7b11f49bf19adc is joining an existing cluster
I20260806 12:23:55.631750 7672 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.210.124:39995
I20260806 12:23:55.636409 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 6876
I20260806 12:23:55.645645 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.635238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.28.210.126:37041)
0806 12:23:55.635397 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.635403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.635858 (+ 455us) server_negotiation.cc:407] Connection header received
0806 12:23:55.636195 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.636199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.636268 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.636328 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.637119 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.637934 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.639160 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.639526 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.639606 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.641166 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.641171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.641174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.641205 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.643384 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.644035 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.644039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.644041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.644163 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.644594 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.644597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.644599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.644824 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.644900 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.645301 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.645477 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 12:23:55.645645 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.635826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.28.210.125:46317)
0806 12:23:55.636251 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.636257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.636918 (+ 661us) server_negotiation.cc:407] Connection header received
0806 12:23:55.637123 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.637127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.637183 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.637236 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.637953 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.638783 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.639953 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.640623 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.640712 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.642089 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.642094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.642099 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.642131 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.643705 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.644193 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.644197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.644199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.644288 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.644586 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.644589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.644591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.644824 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.644900 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.645301 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.645458 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313}
I20260806 12:23:55.646282 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.635187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:37818)
0806 12:23:55.635755 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.635773 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.636018 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.636526 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.636529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.636806 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.636968 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.636977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.638078 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.638081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.639018 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.639027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.639810 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.639814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.639897 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.640497 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.640514 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.640992 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.643610 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.643613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.643615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.643915 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.644318 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.644321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.644323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.644493 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.645025 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.645027 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.645107 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.645909 (+ 802us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.646084 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":25,"spinlock_wait_cycles":2304,"thread_start_us":155,"threads_started":1}
I20260806 12:23:55.646808 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.635415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:46317 (local address 127.0.0.1:43540)
0806 12:23:55.636807 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.636823 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.637006 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.637393 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.637396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.637644 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.637785 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.637792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.638955 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.638959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.639809 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.639816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.640876 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.640879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.641001 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.641466 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.641482 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.641899 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.643833 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.643836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.643838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.644059 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.644395 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.644398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.644400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.644475 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.644994 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.644997 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.645116 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.646301 (+ 1185us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.646627 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":112,"threads_started":1}
I20260806 12:23:55.678966 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.636336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45238 (local address 127.28.210.124:39995)
0806 12:23:55.636449 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.636454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.646553 (+ 10099us) server_negotiation.cc:407] Connection header received
0806 12:23:55.646707 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.646710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.646751 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.646795 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.647483 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.647996 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.648770 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.648999 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.649089 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.650206 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.650210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.650213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.650237 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.651614 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.652012 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.652016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.652018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.652126 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.677562 (+ 25436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.677568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.677570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.677951 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.678076 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.678550 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.678752 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 12:23:55.679663 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.635823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:39995 (local address 127.0.0.1:45238)
0806 12:23:55.646482 (+ 10659us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.646496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.646618 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.646922 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.646926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.647202 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.647350 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.647359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.648115 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.648118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.648653 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.648661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.649108 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.649111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.649195 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.649700 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.649712 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.650062 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.651729 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.651732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.651734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.651922 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.677125 (+ 25203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.677131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.677133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.677346 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.678176 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.678180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.678305 (+ 125us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.679375 (+ 1070us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.679535 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10583,"thread_start_us":129,"threads_started":1}
I20260806 12:23:55.680455 7278 heartbeater.cc:503] Master 127.28.210.126:37041 was elected leader, sending a full tablet report...
I20260806 12:23:55.688952 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:37747
--local_ip_for_outbound_sockets=127.28.210.65
--tserver_master_addrs=127.28.210.126:37041
--webserver_port=44207
--webserver_interface=127.28.210.65
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab
--principal=oryx/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.210.126:37041,127.28.210.125:46317
--tserver_master_addrs=127.28.210.126:37041,127.28.210.125:46317,127.28.210.124:39995 with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019034, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.126@KRBTEST.COM for oryx/127.28.210.124@KRBTEST.COM
I20260806 12:23:55.719245 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.713907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.28.210.125:46317)
0806 12:23:55.714057 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.714062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.714366 (+ 304us) server_negotiation.cc:407] Connection header received
0806 12:23:55.714543 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.714547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.714629 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.714745 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:55.715305 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.716060 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.717907 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.718480 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.718638 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.718858 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.718948 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.719058 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 12:23:55.719889 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.713833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:46317 (local address 127.0.0.1:43552)
0806 12:23:55.714249 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.714267 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.714417 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.714927 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.714931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.714957 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:55.715109 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.715118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.716298 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.716302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.717736 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.717753 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.718638 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.718643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.718760 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.718816 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.719563 (+ 747us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.719702 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":128,"threads_started":1}
I20260806 12:23:55.720880 7730 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260806 12:23:55.721635 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82a2d105a1274cba837b796cd94f697c. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true } } }
I20260806 12:23:55.721858 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: This master's current role is: LEARNER
I20260806 12:23:55.728603 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.686430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:39995 (local address 127.0.0.1:45244)
0806 12:23:55.687586 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.687601 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.687711 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.688445 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.688449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.688688 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.688827 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.688837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.690058 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.690062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.690939 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.690951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.691695 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.691700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.691830 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.692712 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.692732 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.694632 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.726491 (+ 31859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.726495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.726498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.726851 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.727234 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.727238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.727240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.727362 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.727852 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.727856 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.727988 (+ 132us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.728163 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.728374 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079}
I20260806 12:23:55.729375 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.686904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.28.210.124:39995)
0806 12:23:55.687353 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.687358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.687978 (+ 620us) server_negotiation.cc:407] Connection header received
0806 12:23:55.688108 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.688111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.688183 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.688262 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.688980 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.689919 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.691070 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.691419 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.691527 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.694878 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.694883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.694886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.694925 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.726287 (+ 31362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.726998 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.727003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.727005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.727124 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.727462 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.727465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.727468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.727730 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.727840 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.729065 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.729219 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 12:23:55.730690 6796 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.210.124:39995: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc"
member_type: NON_VOTER
last_known_addr {
host: "127.28.210.124"
port: 39995
}
attrs {
promote: true
}
}
dest_uuid: "82a2d105a1274cba837b796cd94f697c"
cas_config_opid_index: 5
I20260806 12:23:55.731518 6796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } }
I20260806 12:23:55.733228 7669 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c0c3343c36004df38a7b11f49bf19adc (127.28.210.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } } }
I20260806 12:23:55.735388 7669 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.210.124:39995
I20260806 12:23:55.735738 7669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } } }
I20260806 12:23:55.735906 7669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:55.736200 7672 master.cc:537] Master@127.28.210.124:39995 shutting down...
I20260806 12:23:55.739696 7730 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 LEARNER]: Refusing update from remote peer 82a2d105a1274cba837b796cd94f697c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 12:23:55.742843 7666 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 12:23:55.746640 7730 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c0c3343c36004df38a7b11f49bf19adc (127.28.210.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } } }
W20260806 12:23:55.748492 6771 consensus_peers.cc:487] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c -> Peer c0c3343c36004df38a7b11f49bf19adc (127.28.210.124:39995): Couldn't send request to peer c0c3343c36004df38a7b11f49bf19adc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 12:23:55.754541 7672 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 0 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:55.754956 7672 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 0 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:55.755131 7672 tablet_replica.cc:330] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: stopping tablet replica
I20260806 12:23:55.758436 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: NON_VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: true } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } } }
I20260806 12:23:55.758675 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: This master's current role is: LEARNER
I20260806 12:23:55.759841 7669 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c: attempting to promote NON_VOTER 3c7a6f38f9fc40f187bf1ef7334aca6a to VOTER
I20260806 12:23:55.760672 7669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } }
I20260806 12:23:55.762692 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.762367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:39995 (local address 127.0.0.1:45252)
0806 12:23:55.762496 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.762587 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:39995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 12:23:55.763747 7730 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 LEARNER]: Refusing update from remote peer 82a2d105a1274cba837b796cd94f697c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 12:23:55.764029 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.763785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:39995 (local address 127.0.0.1:45256)
0806 12:23:55.763896 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.763943 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:39995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 12:23:55.764361 6771 consensus_peers.cc:487] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c -> Peer c0c3343c36004df38a7b11f49bf19adc (127.28.210.124:39995): Couldn't send request to peer c0c3343c36004df38a7b11f49bf19adc. Status: Network error: Client connection negotiation failed: client connection to 127.28.210.124:39995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 12:23:55.764695 7940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 12:23:55.767539 7666 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c7a6f38f9fc40f187bf1ef7334aca6a (127.28.210.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } } }
I20260806 12:23:55.770551 7669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } } }
I20260806 12:23:55.770776 7669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:55.771255 7669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } } }
I20260806 12:23:55.771401 7669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:55.772513 7730 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c7a6f38f9fc40f187bf1ef7334aca6a (127.28.210.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } } }
I20260806 12:23:55.775388 7672 master.cc:552] Master@127.28.210.124:39995 shutdown complete.
I20260806 12:23:55.775468 7672 master_runner.cc:305] Clearing existing system tablet
I20260806 12:23:55.775782 7672 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:55.775863 7672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:55.778085 7672 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260806 12:23:55.779217 7947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:55.779683 7672 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:55.779837 7672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 12:23:55.779881 7672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/wal
uuid: "c0c3343c36004df38a7b11f49bf19adc"
format_stamp: "Formatted at 2026-08-06 12:23:55 on dist-test-slave-lgr5"
server_key: "002d080e7562e7a207c65160aa5462c9"
server_key_iv: "472e10ed7965ac1bfe139680fd0be7b1"
server_key_version: "encryptionkey@0"
I20260806 12:23:55.782902 7672 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:55.785779 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } } }
I20260806 12:23:55.786038 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:23:55.788321 7672 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 12:23:55.788466 7672 log.cc:1196] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 12:23:55.788803 7672 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: Deleting consensus metadata
I20260806 12:23:55.789028 7672 master_runner.cc:315] Copying system tablet from 127.28.210.126:37041
I20260806 12:23:55.790584 7672 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: tablet copy: Beginning tablet copy session from remote peer at address 127.28.210.126:37041
I20260806 12:23:55.800592 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.791263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:37822)
0806 12:23:55.792358 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.792371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.792536 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.792817 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.792820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.793067 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.793462 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.793470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.794453 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.794457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.795112 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.795123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.795661 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.795666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.795780 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.796499 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.796514 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.796985 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.798829 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.798833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.798836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.799058 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.799365 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.799369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.799371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.799469 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.799892 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.799895 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.800009 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.800185 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.800360 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"thread_start_us":97,"threads_started":1}
I20260806 12:23:55.800637 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.791344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.28.210.126:37041)
0806 12:23:55.791487 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.791492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.792468 (+ 976us) server_negotiation.cc:407] Connection header received
0806 12:23:55.792642 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.792645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.792704 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.792785 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.793712 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.794317 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.795249 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.795505 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.795639 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.797173 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.797177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.797179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.797208 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.798700 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.799162 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.799166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.799169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.799262 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.799563 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.799566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.799568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.799789 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.799892 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.800361 (+ 469us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.800490 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:23:55.801460 6823 tablet_copy_service.cc:140] P 82a2d105a1274cba837b796cd94f697c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c0c3343c36004df38a7b11f49bf19adc ({username='oryx', principal='oryx/127.28.210.124@KRBTEST.COM'} at 127.0.0.1:37822)
I20260806 12:23:55.801577 6823 tablet_copy_service.cc:161] P 82a2d105a1274cba837b796cd94f697c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c0c3343c36004df38a7b11f49bf19adc at {username='oryx', principal='oryx/127.28.210.124@KRBTEST.COM'} at 127.0.0.1:37822: session id = c0c3343c36004df38a7b11f49bf19adc-00000000000000000000000000000000
I20260806 12:23:55.802580 6823 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 12:23:55.803967 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 12:23:55.806497 7672 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: tablet copy: Starting download of 0 data blocks...
I20260806 12:23:55.806715 7672 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: tablet copy: Starting download of 1 WAL segments...
I20260806 12:23:55.808499 7672 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 12:23:55.810662 6823 tablet_copy_service.cc:342] P 82a2d105a1274cba837b796cd94f697c: Request end of tablet copy session c0c3343c36004df38a7b11f49bf19adc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.210.124@KRBTEST.COM'} at 127.0.0.1:37822
I20260806 12:23:55.810782 6823 tablet_copy_service.cc:433] P 82a2d105a1274cba837b796cd94f697c: ending tablet copy session c0c3343c36004df38a7b11f49bf19adc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c0c3343c36004df38a7b11f49bf19adc
I20260806 12:23:55.819486 7672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:55.821753 7963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:55.822656 7672 server_base.cc:1034] running on GCE node
W20260806 12:23:55.824306 7956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:55.828333 7961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:55.828747 7672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:55.829154 7672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:55.830385 7672 hybrid_clock.cc:648] HybridClock initialized: now 1786019035830370 us; error 39 us; skew 500 ppm
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:55.832008 7672 init.cc:377] Logged in from keytab as oryx/127.28.210.124@KRBTEST.COM (short username oryx)
I20260806 12:23:55.832638 7672 webserver.cc:460] Webserver started at http://127.28.210.124:34509/ using document root <none> and password file <none>
I20260806 12:23:55.832851 7672 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:55.832898 7672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:55.834427 7672 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 12:23:55.835549 7972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:55.835844 7672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 12:23:55.835928 7672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/wal
uuid: "c0c3343c36004df38a7b11f49bf19adc"
format_stamp: "Formatted at 2026-08-06 12:23:55 on dist-test-slave-lgr5"
server_key: "002d080e7562e7a207c65160aa5462c9"
server_key_iv: "472e10ed7965ac1bfe139680fd0be7b1"
server_key_version: "encryptionkey@0"
I20260806 12:23:55.836028 7672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:55.842415 7672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:55.844901 7672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:55.849855 7672 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.124:39995
I20260806 12:23:55.850751 7672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
I20260806 12:23:55.850915 8042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.124:39995 every 8 connection(s)
I20260806 12:23:55.852612 8044 sys_catalog.cc:263] Verifying existing consensus state
I20260806 12:23:55.853652 8044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: Bootstrap starting.
I20260806 12:23:55.864828 8044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 12:23:55.865167 8044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: Bootstrap complete.
I20260806 12:23:55.865777 8044 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } }
I20260806 12:23:55.865938 8044 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c0c3343c36004df38a7b11f49bf19adc, State: Initialized, Role: LEARNER
I20260806 12:23:55.866045 8044 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } }
I20260806 12:23:55.866588 8044 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:23:55.866504 8048 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } } }
I20260806 12:23:55.866676 8048 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: This master's current role is: LEARNER
I20260806 12:23:55.870555 8066 catalog_manager.cc:1259] Loaded cluster ID: 8165e75ab35f4f5aadc5653b708b0362
I20260806 12:23:55.870632 8066 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: loading cluster ID for follower catalog manager: success
I20260806 12:23:55.872426 8066 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: acquiring CA information for follower catalog manager: success
I20260806 12:23:55.873873 8066 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 12:23:55.878296 7936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:55.878634 7936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:55.878680 7936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:55.883081 7936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:55.883180 7936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:55.883245 7936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:23:55.889217 7936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab
--principal=oryx/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:37747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=44207
--tserver_master_addrs=127.28.210.126:37041,127.28.210.125:46317,127.28.210.124:39995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:55.890576 7936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:55.891770 7936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:55.902819 8074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:55.902865 7936 server_base.cc:1034] running on GCE node
W20260806 12:23:55.902808 8079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:55.902938 8076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:55.903688 7936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:55.904541 7936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:55.905755 7936 hybrid_clock.cc:648] HybridClock initialized: now 1786019035905717 us; error 64 us; skew 500 ppm
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:55.909178 7936 init.cc:377] Logged in from keytab as oryx/127.28.210.65@KRBTEST.COM (short username oryx)
I20260806 12:23:55.910310 7936 webserver.cc:460] Webserver started at http://127.28.210.65:44207/ using document root <none> and password file <none>
I20260806 12:23:55.910959 7936 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:55.911063 7936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:55.916122 7936 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260806 12:23:55.918777 8088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:55.920491 7936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:55.920709 7936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "459f935f051e42d4938e20b9e58425b2"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "77332d99401f6e0e1bd942714c9dc1c6"
server_key_iv: "ce89de5ba50913462c1fcee0794ecc0d"
server_key_version: "encryptionkey@0"
I20260806 12:23:55.921486 7936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:55.942440 7936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:55.946610 7936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:55.946822 7936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:55.947525 7936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:55.948693 7936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:55.948750 7936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:55.948796 7936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:55.948810 7936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:55.949573 7504 heartbeater.cc:503] Master 127.28.210.126:37041 was elected leader, sending a full tablet report...
I20260806 12:23:55.961918 7936 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:37747
I20260806 12:23:55.962037 8266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:37747 every 8 connection(s)
I20260806 12:23:55.963205 7936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:23:55.965715 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 7936
I20260806 12:23:55.965951 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 7091
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.125@KRBTEST.COM
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:23:55.987246 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.965895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:52187 (local address 127.28.210.125:46317)
0806 12:23:55.966233 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.966239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.969716 (+ 3477us) server_negotiation.cc:407] Connection header received
0806 12:23:55.971123 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.971128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.971201 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.971258 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.973453 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.974563 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.976714 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.977043 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.977119 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.982061 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.982066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.982069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.982107 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.984139 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.984949 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.984953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.984955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.985063 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.985610 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.985614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.985616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.985904 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.986006 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.986902 (+ 896us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.987057 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260806 12:23:55.987370 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:41305
--local_ip_for_outbound_sockets=127.28.210.66
--tserver_master_addrs=127.28.210.126:37041
--webserver_port=40255
--webserver_interface=127.28.210.66
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab
--principal=oryx/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.210.126:37041,127.28.210.125:46317
--tserver_master_addrs=127.28.210.126:37041,127.28.210.125:46317,127.28.210.124:39995 with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:23:55.989223 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.966345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:46317 (local address 127.28.210.65:52187)
0806 12:23:55.969492 (+ 3147us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.969581 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.970817 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.971746 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.971756 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.972345 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.973170 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.973200 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.974837 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.974841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.976532 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.976550 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.977368 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.977372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.977614 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.978996 (+ 1382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.979024 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.981641 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.984298 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.984306 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.984331 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.984781 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.985225 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.985228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.985230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.985471 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.986091 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.986100 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.986575 (+ 475us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.987449 (+ 874us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.987905 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2835,"thread_start_us":140,"threads_started":1}
I20260806 12:23:55.989897 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.966075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:41343 (local address 127.28.210.126:37041)
0806 12:23:55.966327 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.966332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.979361 (+ 13029us) server_negotiation.cc:407] Connection header received
0806 12:23:55.979608 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.979612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.979674 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.979743 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.980506 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.981302 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.982833 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.983206 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.983303 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.986110 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.986114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.986118 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.986153 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.987949 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.988493 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.988497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.988498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.988597 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.988920 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.988924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.988926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.989161 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.989245 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.989594 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.989739 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260806 12:23:55.990483 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.966345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.65:41343)
0806 12:23:55.979212 (+ 12867us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.979231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.979470 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.979924 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.979927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.980194 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.980344 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.980354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.981642 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.981646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.982660 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.982678 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.983488 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.983491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.983570 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.984186 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.984203 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.985904 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.988132 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.988136 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.988138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.988385 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.988704 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.988707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.988709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.988796 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.989304 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.989307 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.989405 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.990106 (+ 701us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.990308 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12703,"spinlock_wait_cycles":57088,"thread_start_us":257,"threads_started":1}
I20260806 12:23:55.990981 8271 heartbeater.cc:347] Connected to a master server at 127.28.210.125:46317
I20260806 12:23:55.991194 8270 heartbeater.cc:347] Connected to a master server at 127.28.210.126:37041
I20260806 12:23:55.991258 8271 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:55.991259 8270 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:55.991979 8270 heartbeater.cc:511] Master 127.28.210.126:37041 requested a full tablet report, sending...
I20260806 12:23:55.992065 8271 heartbeater.cc:511] Master 127.28.210.125:46317 requested a full tablet report, sending...
Aug 06 12:23:55 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019035, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.124@KRBTEST.COM
I20260806 12:23:55.993491 6796 ts_manager.cc:194] Re-registered known tserver with Master: 459f935f051e42d4938e20b9e58425b2 (127.28.210.65:37747)
I20260806 12:23:55.993784 7717 ts_manager.cc:194] Registered new tserver with Master: 459f935f051e42d4938e20b9e58425b2 (127.28.210.65:37747)
I20260806 12:23:55.994289 6796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:41343
I20260806 12:23:55.996903 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.969432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:47113 (local address 127.28.210.124:39995)
0806 12:23:55.984011 (+ 14579us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:55.984017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:55.984433 (+ 416us) server_negotiation.cc:407] Connection header received
0806 12:23:55.984640 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:55.984644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:55.984705 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:55.984777 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:55.985523 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.986547 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.988493 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.988846 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.988931 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.992828 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:55.992833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:55.992835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:55.992881 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:55.994711 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.995361 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.995366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.995368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.995472 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:55.995810 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:55.995813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:55.995815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:55.996131 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:55.996226 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:55.996622 (+ 396us) server_negotiation.cc:299] Negotiation successful
0806 12:23:55.996750 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14453,"thread_start_us":1518,"threads_started":1}
I20260806 12:23:55.996922 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:55.967647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:39995 (local address 127.28.210.65:47113)
0806 12:23:55.984323 (+ 16676us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:55.984340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:55.984523 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:55.984962 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:55.984965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:55.985144 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:55.985367 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.985379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.986844 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.986848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.988335 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:55.988345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:55.989131 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:55.989135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:55.989229 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:55.990069 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:55.990085 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:55.992633 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:55.994894 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.994897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.994899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.995206 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.995596 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:55.995600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:55.995601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:55.995697 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:55.996294 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:55.996297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:55.996400 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 12:23:55.996508 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 12:23:55.996695 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16598,"thread_start_us":1390,"threads_started":1}
I20260806 12:23:55.998109 8268 heartbeater.cc:347] Connected to a master server at 127.28.210.124:39995
I20260806 12:23:55.998242 8268 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:55.998534 8268 heartbeater.cc:511] Master 127.28.210.124:39995 requested a full tablet report, sending...
I20260806 12:23:56.000036 7992 ts_manager.cc:194] Registered new tserver with Master: 459f935f051e42d4938e20b9e58425b2 (127.28.210.65:37747)
W20260806 12:23:56.126070 8279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:56.126438 8279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:56.126515 8279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:56.131626 8279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:56.131736 8279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:56.131803 8279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:23:56.137813 8279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab
--principal=oryx/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:41305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=40255
--tserver_master_addrs=127.28.210.126:37041,127.28.210.125:46317,127.28.210.124:39995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:56.139041 8279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:56.140205 8279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:56.150751 8289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:56.150870 8279 server_base.cc:1034] running on GCE node
W20260806 12:23:56.151036 8287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:56.151141 8286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:56.151686 8279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:56.152570 8279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:56.153769 8279 hybrid_clock.cc:648] HybridClock initialized: now 1786019036153741 us; error 38 us; skew 500 ppm
Aug 06 12:23:56 dist-test-slave-lgr5 krb5kdc[6725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019036, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:56.157684 8279 init.cc:377] Logged in from keytab as oryx/127.28.210.66@KRBTEST.COM (short username oryx)
I20260806 12:23:56.159135 8279 webserver.cc:460] Webserver started at http://127.28.210.66:40255/ using document root <none> and password file <none>
I20260806 12:23:56.160010 8279 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:56.160175 8279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:56.166736 8279 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 12:23:56.169667 8298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:56.171427 8279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 12:23:56.171593 8279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "55b89e5769ac4aeabaa157a85d382d85"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "15995113bde3c4b3b84548910822f6bc"
server_key_iv: "ef3ce55b7a928023efc5b35826cf1f63"
server_key_version: "encryptionkey@0"
I20260806 12:23:56.172359 8279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:56.196333 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.192182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:39995 (local address 127.0.0.1:45262)
0806 12:23:56.192336 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.192353 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.192458 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.192784 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.192787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.192805 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:56.192899 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.192904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.193946 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.193950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.194730 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.194740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.195856 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.195859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.195943 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.195970 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.196102 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.196198 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 12:23:56.196362 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.192270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.28.210.124:39995)
0806 12:23:56.192440 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.192444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.192464 (+ 20us) server_negotiation.cc:407] Connection header received
0806 12:23:56.192570 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.192573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.192646 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.192754 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:56.192989 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.193778 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.194906 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.195687 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.195855 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.196053 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.196132 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.196231 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 12:23:56.197325 8006 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 12:23:56.197782 8048 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82a2d105a1274cba837b796cd94f697c. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } } }
I20260806 12:23:56.197959 8048 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: This master's current role is: LEARNER
I20260806 12:23:56.199400 8279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:56.202999 8279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:56.203243 8279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:56.203999 8279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:56.205217 8279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:56.205274 8279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:56.205354 8279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:56.205384 8279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:56.218990 8279 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:41305
I20260806 12:23:56.219015 8484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:41305 every 8 connection(s)
I20260806 12:23:56.220507 8279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:23:56.228673 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 8279
I20260806 12:23:56.228881 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 7289
Aug 06 12:23:56 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019036, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.124@KRBTEST.COM
Aug 06 12:23:56 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019036, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.125@KRBTEST.COM
I20260806 12:23:56.240973 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.224586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:37543 (local address 127.28.210.124:39995)
0806 12:23:56.224774 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.224779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.225492 (+ 713us) server_negotiation.cc:407] Connection header received
0806 12:23:56.225803 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.225806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.225863 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.225915 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.227853 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.228884 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.231254 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.231734 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.231865 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.235759 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.235776 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.235780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.235820 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.237653 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.238478 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.238482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.238484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.238594 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.239102 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.239106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.239108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.239370 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.239451 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.240614 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.240794 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 12:23:56.241284 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.225763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:36401 (local address 127.28.210.125:46317)
0806 12:23:56.225964 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.225969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.225985 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:23:56.226121 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.226125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.226175 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.226241 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.228323 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.229088 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.230932 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.231240 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.231311 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.235814 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.235820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.235823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.235866 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.238589 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.239342 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.239347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.239350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.239474 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.239925 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.239930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.239933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.240264 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.240364 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.240810 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.241095 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260806 12:23:56.242710 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.225031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:39995 (local address 127.28.210.66:37543)
0806 12:23:56.225364 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.225378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.225651 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.226237 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.226248 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.226920 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:56.227656 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.227668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.229116 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.229120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.231030 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.231049 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.232129 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.232135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.232253 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.232764 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:56.232807 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:56.235446 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:56.237869 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:56.237878 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:56.237903 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:56.238335 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:56.238726 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:56.238729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:56.238731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:56.238975 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:56.239547 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:56.239556 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:56.240174 (+ 618us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.241191 (+ 1017us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.241621 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":175,"spinlock_wait_cycles":39808,"thread_start_us":120,"threads_started":1}
I20260806 12:23:56.242738 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.225448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:46317 (local address 127.28.210.66:36401)
0806 12:23:56.225721 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.225735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.226019 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.226273 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.226276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.226894 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:56.227647 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.227666 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.229239 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.229243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.230691 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.230707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.231539 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.231543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.231757 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.232777 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:56.232799 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:56.235447 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:56.238802 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:56.238806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:56.238809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:56.239173 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:56.239630 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:56.239634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:56.239637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:56.239784 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:56.240458 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:56.240461 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:56.240578 (+ 117us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.242216 (+ 1638us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.242440 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":101,"spinlock_wait_cycles":9984,"thread_start_us":97,"threads_started":1}
I20260806 12:23:56.261193 8486 heartbeater.cc:347] Connected to a master server at 127.28.210.124:39995
I20260806 12:23:56.261634 8486 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:56.261212 8489 heartbeater.cc:347] Connected to a master server at 127.28.210.125:46317
I20260806 12:23:56.261878 8489 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:56.262359 8489 heartbeater.cc:511] Master 127.28.210.125:46317 requested a full tablet report, sending...
Aug 06 12:23:56 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019036, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:23:56.262356 8486 heartbeater.cc:511] Master 127.28.210.124:39995 requested a full tablet report, sending...
I20260806 12:23:56.264070 7992 ts_manager.cc:194] Registered new tserver with Master: 55b89e5769ac4aeabaa157a85d382d85 (127.28.210.66:41305)
I20260806 12:23:56.264070 7717 ts_manager.cc:194] Registered new tserver with Master: 55b89e5769ac4aeabaa157a85d382d85 (127.28.210.66:41305)
I20260806 12:23:56.267629 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.222809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:51953 (local address 127.28.210.126:37041)
0806 12:23:56.222956 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.222960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.224236 (+ 1276us) server_negotiation.cc:407] Connection header received
0806 12:23:56.225538 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.225541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.225611 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.225677 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.227852 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.228920 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.258870 (+ 29950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.259224 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.259327 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.263212 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.263217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.263220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.263259 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.265598 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.266134 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.266138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.266140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.266253 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.266674 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.266678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.266679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.266906 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.267010 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.267326 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.267455 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 12:23:56.267656 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.223232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.66:51953)
0806 12:23:56.223963 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.224016 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.225186 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.226237 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.226248 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.226911 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:56.227654 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.227668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.257751 (+ 30083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.257758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.258617 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.258638 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.259599 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.259604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.259727 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.260478 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:56.260496 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:56.262939 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:56.265761 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:56.265764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:56.265766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:56.265995 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:56.266429 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:56.266433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:56.266435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:56.266551 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:56.267019 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:56.267022 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:56.267141 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.267227 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.267420 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":103,"spinlock_wait_cycles":3328,"thread_start_us":295,"threads_started":1}
I20260806 12:23:56.268656 8488 heartbeater.cc:347] Connected to a master server at 127.28.210.126:37041
I20260806 12:23:56.268774 8488 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:56.268997 8488 heartbeater.cc:511] Master 127.28.210.126:37041 requested a full tablet report, sending...
I20260806 12:23:56.269774 6796 ts_manager.cc:194] Re-registered known tserver with Master: 55b89e5769ac4aeabaa157a85d382d85 (127.28.210.66:41305)
I20260806 12:23:56.270401 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:33181
--local_ip_for_outbound_sockets=127.28.210.67
--tserver_master_addrs=127.28.210.126:37041
--webserver_port=46563
--webserver_interface=127.28.210.67
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab
--principal=oryx/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.210.126:37041,127.28.210.125:46317
--tserver_master_addrs=127.28.210.126:37041,127.28.210.125:46317,127.28.210.124:39995 with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:23:56.270432 6796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:51953
W20260806 12:23:56.403221 8496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:23:56.403550 8496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:23:56.403615 8496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:23:56.407984 8496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:23:56.408102 8496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:23:56.408190 8496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:23:56.414101 8496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab
--principal=oryx/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:33181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=46563
--tserver_master_addrs=127.28.210.126:37041,127.28.210.125:46317,127.28.210.124:39995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:23:56.415448 8496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:23:56.416612 8496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:23:56.428427 8507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:56.428705 8496 server_base.cc:1034] running on GCE node
W20260806 12:23:56.428757 8505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:23:56.428707 8509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:23:56.429513 8496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:23:56.430416 8496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:23:56.431599 8496 hybrid_clock.cc:648] HybridClock initialized: now 1786019036431576 us; error 34 us; skew 500 ppm
Aug 06 12:23:56 dist-test-slave-lgr5 krb5kdc[6725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019036, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:23:56.435202 8496 init.cc:377] Logged in from keytab as oryx/127.28.210.67@KRBTEST.COM (short username oryx)
I20260806 12:23:56.436499 8496 webserver.cc:460] Webserver started at http://127.28.210.67:46563/ using document root <none> and password file <none>
I20260806 12:23:56.437083 8496 fs_manager.cc:359] Metadata directory not provided
I20260806 12:23:56.437160 8496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:23:56.441422 8496 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:23:56.443797 8525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:56.445395 8496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:23:56.445569 8496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "2e87c17242b84502bb0db4ad9d7cf416"
format_stamp: "Formatted at 2026-08-06 12:23:54 on dist-test-slave-lgr5"
server_key: "1df665410c58a0b8fa76335ba61b71f6"
server_key_iv: "6ccc2b7417c1ca8b1ab6b7697e4b097e"
server_key_version: "encryptionkey@0"
I20260806 12:23:56.446168 8496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:23:56.465241 8496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:23:56.469311 8496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:23:56.469527 8496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:23:56.470242 8496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:23:56.471264 8496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:23:56.471340 8496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:56.471385 8496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:23:56.471412 8496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:23:56.483163 8496 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:33181
I20260806 12:23:56.483184 8698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:33181 every 8 connection(s)
I20260806 12:23:56.484665 8496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:23:56.489753 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 8496
Aug 06 12:23:56 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019036, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.125@KRBTEST.COM
Aug 06 12:23:56 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019036, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.124@KRBTEST.COM
Aug 06 12:23:56 dist-test-slave-lgr5 krb5kdc[6725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019036, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:23:56.505085 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.487453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:33949 (local address 127.28.210.125:46317)
0806 12:23:56.487611 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.487616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.488954 (+ 1338us) server_negotiation.cc:407] Connection header received
0806 12:23:56.490601 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.490605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.490693 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.490754 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.492946 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.493742 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.495717 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.496047 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.496176 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.500185 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.500190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.500192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.500224 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.502311 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.502968 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.502972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.502974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.503064 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.503523 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.503527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.503529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.503774 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.503855 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.504737 (+ 882us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.504873 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 12:23:56.507010 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.487453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:36875 (local address 127.28.210.126:37041)
0806 12:23:56.487610 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.487616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.488994 (+ 1378us) server_negotiation.cc:407] Connection header received
0806 12:23:56.490575 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.490578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.490640 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.490693 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.493234 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.494037 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.498380 (+ 4343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.498717 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.498828 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.502042 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.502046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.502049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.502080 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.504971 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.505537 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.505541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.505543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.505642 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.505971 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.505975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.505977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.506225 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.506315 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.506709 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.506846 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 12:23:56.508221 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.488016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.67:36875)
0806 12:23:56.488875 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.488891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.490291 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.491155 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.491165 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.492959 (+ 1794us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:56.493079 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.493087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.494176 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.494179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.498266 (+ 4087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.498280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.499463 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.499468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.499577 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.500342 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:56.500357 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:56.501845 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:56.505180 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:56.505183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:56.505185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:56.505423 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:56.505738 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:56.505742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:56.505744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:56.505839 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:56.506382 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:56.506384 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:56.506503 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.507189 (+ 686us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.507352 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":3018,"spinlock_wait_cycles":31872,"thread_start_us":156,"threads_started":1}
I20260806 12:23:56.509083 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.489542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:60551 (local address 127.28.210.124:39995)
0806 12:23:56.489672 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.489676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.490102 (+ 426us) server_negotiation.cc:407] Connection header received
0806 12:23:56.490574 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.490578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.490640 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.490693 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.493459 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.494336 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.495908 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.496367 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.496475 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.501193 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.501199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.501202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.501239 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.506078 (+ 4839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.506613 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.506616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.506618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.506708 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.507004 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.507008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.507010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.507231 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.507316 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.508784 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.508908 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 12:23:56.509168 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.492185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.28.210.126:37041)
0806 12:23:56.492439 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.492443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.492881 (+ 438us) server_negotiation.cc:407] Connection header received
0806 12:23:56.493112 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.493115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.493168 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.493349 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.494461 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.495266 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.496816 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.497134 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.497215 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.499873 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.499878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.499881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.499908 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.502375 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.506773 (+ 4398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.506777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.506779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.506882 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.507775 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.507779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.507781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.508015 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.508167 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.508815 (+ 648us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.508985 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260806 12:23:56.509171 8709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.489395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:39995 (local address 127.28.210.67:60551)
0806 12:23:56.489870 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.489883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.490295 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.491155 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.491165 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.491828 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:56.492731 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.492758 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.494836 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.494840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.495702 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.495715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.496670 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.496675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.496797 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.498017 (+ 1220us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:56.498035 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:56.501036 (+ 3001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:56.506226 (+ 5190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:56.506229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:56.506231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:56.506491 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:56.506813 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:56.506816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:56.506818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:56.506895 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:56.507360 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:56.507363 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:56.507466 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.507558 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.507700 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":43,"spinlock_wait_cycles":4480,"thread_start_us":164,"threads_started":1}
I20260806 12:23:56.509979 8708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.488016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:46317 (local address 127.28.210.67:33949)
0806 12:23:56.488737 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.488804 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.490304 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.491157 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.491165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.491796 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:23:56.492743 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.492771 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.493938 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.493941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.495572 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.495585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.496400 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.496404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.496638 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.497450 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:23:56.497477 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:23:56.499928 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:23:56.502457 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:56.502465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:56.502486 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:56.502847 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:56.503177 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:23:56.503180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:23:56.503182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:23:56.503384 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:23:56.503949 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:23:56.503956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:23:56.504389 (+ 433us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.505314 (+ 925us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.506034 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":91,"spinlock_wait_cycles":13312,"thread_start_us":202,"threads_started":1}
I20260806 12:23:56.510860 8700 heartbeater.cc:347] Connected to a master server at 127.28.210.124:39995
I20260806 12:23:56.511157 8700 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:56.511258 8702 heartbeater.cc:347] Connected to a master server at 127.28.210.125:46317
I20260806 12:23:56.511363 8702 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:56.511629 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.495441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.28.210.124:39995)
0806 12:23:56.498850 (+ 3409us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.498855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.498873 (+ 18us) server_negotiation.cc:407] Connection header received
0806 12:23:56.498921 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.498924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.498983 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.499228 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.502264 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.503065 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.504145 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.504473 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.504577 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.506846 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.506850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.506852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.506879 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.508594 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.509049 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.509053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.509054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.509139 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.510015 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.510019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.510021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.510266 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.510397 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.511165 (+ 768us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.511473 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3298,"thread_start_us":117,"threads_started":1}
I20260806 12:23:56.510865 8701 heartbeater.cc:347] Connected to a master server at 127.28.210.126:37041
I20260806 12:23:56.511746 8701 heartbeater.cc:464] Registering TS with master...
I20260806 12:23:56.511854 8700 heartbeater.cc:511] Master 127.28.210.124:39995 requested a full tablet report, sending...
I20260806 12:23:56.511904 8701 heartbeater.cc:511] Master 127.28.210.126:37041 requested a full tablet report, sending...
I20260806 12:23:56.513041 8702 heartbeater.cc:511] Master 127.28.210.125:46317 requested a full tablet report, sending...
I20260806 12:23:56.513324 7992 ts_manager.cc:194] Registered new tserver with Master: 2e87c17242b84502bb0db4ad9d7cf416 (127.28.210.67:33181)
I20260806 12:23:56.513446 6796 ts_manager.cc:194] Re-registered known tserver with Master: 2e87c17242b84502bb0db4ad9d7cf416 (127.28.210.67:33181)
I20260806 12:23:56.514302 6796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:36875
I20260806 12:23:56.514335 7717 ts_manager.cc:194] Registered new tserver with Master: 2e87c17242b84502bb0db4ad9d7cf416 (127.28.210.67:33181)
I20260806 12:23:56.516136 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.494668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.28.210.125:46317)
0806 12:23:56.496491 (+ 1823us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.496496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.496512 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:23:56.496556 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.496559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.496611 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.497750 (+ 1139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.497884 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.500138 (+ 2254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.502064 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.502436 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.502950 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.504296 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.504300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.504303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.504329 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.513612 (+ 9283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.514590 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.514594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.514596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.514697 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.515087 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.515090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.515092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.515320 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.515453 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.515771 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.515917 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1738,"thread_start_us":240,"threads_started":1}
I20260806 12:23:56.515241 6796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:23:56.519098 6796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:56.522275 7940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: NON_VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: true } } }
I20260806 12:23:56.522465 7940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:56.527098 8729 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c: attempting to promote NON_VOTER c0c3343c36004df38a7b11f49bf19adc to VOTER
I20260806 12:23:56.527604 8729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } }
I20260806 12:23:56.532383 7730 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 FOLLOWER]: Refusing update from remote peer 82a2d105a1274cba837b796cd94f697c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 12:23:56.532567 8005 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 1 LEARNER]: Refusing update from remote peer 82a2d105a1274cba837b796cd94f697c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 12:23:56.532970 7940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 12:23:56.533361 8726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 12:23:56.536612 8729 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c0c3343c36004df38a7b11f49bf19adc (127.28.210.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } } }
I20260806 12:23:56.537333 8006 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c0c3343c36004df38a7b11f49bf19adc (127.28.210.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } } }
I20260806 12:23:56.538363 7940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } } }
I20260806 12:23:56.538420 8729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } } }
I20260806 12:23:56.538503 7940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:56.538552 8729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:56.538817 8729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } } }
I20260806 12:23:56.538952 8729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:56.539417 7730 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c0c3343c36004df38a7b11f49bf19adc (127.28.210.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } } }
I20260806 12:23:56.539975 8048 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } } }
I20260806 12:23:56.540140 8048 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:23:56.546293 8750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82a2d105a1274cba837b796cd94f697c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } } }
I20260806 12:23:56.546531 8750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:23:56.547652 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.541879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:33181 (local address 127.0.0.1:57510)
0806 12:23:56.542026 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.542047 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.542189 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.543183 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.543188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.543209 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:56.543342 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.543351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.544884 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.544888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.545998 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.546010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.547194 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.547202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.547301 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.547327 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.547398 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.547503 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 12:23:56.548199 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.542129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.28.210.67:33181)
0806 12:23:56.542568 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.542581 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.542606 (+ 25us) server_negotiation.cc:407] Connection header received
0806 12:23:56.542687 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.542692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.542948 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.543234 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:56.543454 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.544774 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.546161 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.547007 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.547181 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.547366 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.547877 (+ 511us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.548021 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":177,"threads_started":1}
I20260806 12:23:56.550932 8745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.542578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:37747 (local address 127.0.0.1:60204)
0806 12:23:56.543682 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.543698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.543924 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.544766 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.544769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.544789 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:56.546233 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.546241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.548037 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.548041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.549288 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.549301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.550450 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.550454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.550547 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.550577 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.550652 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.550740 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":63,"thread_start_us":218,"threads_started":1}
I20260806 12:23:56.550999 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.543460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.28.210.65:37747)
0806 12:23:56.543812 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.543819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.543923 (+ 104us) server_negotiation.cc:407] Connection header received
0806 12:23:56.544156 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.544164 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.544411 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.544580 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:56.546385 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.547888 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.549454 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.550303 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.550429 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.550592 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.550686 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.550788 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":144,"threads_started":1}
I20260806 12:23:56.551244 8749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.543881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:41305 (local address 127.0.0.1:45652)
0806 12:23:56.544230 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.544246 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.544428 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.546396 (+ 1968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.546400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.546422 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:56.546511 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.546655 (+ 144us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.548037 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.548041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.549650 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.549659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.550861 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.550865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.550952 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.550977 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.551046 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.551121 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":23,"thread_start_us":126,"threads_started":1}
I20260806 12:23:56.551679 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.542610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.28.210.66:41305)
0806 12:23:56.545629 (+ 3019us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.545641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.545669 (+ 28us) server_negotiation.cc:407] Connection header received
0806 12:23:56.545757 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.545764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.546011 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.546215 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:56.546787 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.547899 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.549781 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.550541 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.550652 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.550796 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.551456 (+ 660us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.551560 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2894,"thread_start_us":358,"threads_started":1}
I20260806 12:23:56.551853 8605 tablet_service.cc:1467] Processing CreateTablet for tablet bde8d06744444610adbc8636194f9be0 (DEFAULT_TABLE table=test-table [id=d5ef3609ced54ee1b2147ec08ce737f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:56.553236 8605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde8d06744444610adbc8636194f9be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:56.554272 8161 tablet_service.cc:1467] Processing CreateTablet for tablet bde8d06744444610adbc8636194f9be0 (DEFAULT_TABLE table=test-table [id=d5ef3609ced54ee1b2147ec08ce737f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:56.555208 8378 tablet_service.cc:1467] Processing CreateTablet for tablet bde8d06744444610adbc8636194f9be0 (DEFAULT_TABLE table=test-table [id=d5ef3609ced54ee1b2147ec08ce737f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:56.555598 8161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde8d06744444610adbc8636194f9be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:56.556140 8378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde8d06744444610adbc8636194f9be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:56.564419 8753 tablet_bootstrap.cc:492] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85: Bootstrap starting.
I20260806 12:23:56.565038 8754 tablet_bootstrap.cc:492] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416: Bootstrap starting.
I20260806 12:23:56.566902 8756 tablet_bootstrap.cc:492] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2: Bootstrap starting.
I20260806 12:23:56.568867 8753 tablet_bootstrap.cc:654] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:56.569979 8754 tablet_bootstrap.cc:654] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:56.569988 8756 tablet_bootstrap.cc:654] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:56.569999 8753 log.cc:824] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:56.571414 8754 log.cc:824] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:56.571559 8756 log.cc:824] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2: Log is configured to *not* fsync() on all Append() calls
I20260806 12:23:56.573181 8753 tablet_bootstrap.cc:492] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85: No bootstrap required, opened a new log
I20260806 12:23:56.573446 8753 ts_tablet_manager.cc:1397] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260806 12:23:56.574050 8756 tablet_bootstrap.cc:492] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2: No bootstrap required, opened a new log
I20260806 12:23:56.574273 8756 ts_tablet_manager.cc:1397] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260806 12:23:56.580323 8756 raft_consensus.cc:348] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } } peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } }
I20260806 12:23:56.580323 8753 raft_consensus.cc:348] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } } peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } }
I20260806 12:23:56.580726 8753 raft_consensus.cc:374] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:56.580726 8756 raft_consensus.cc:374] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:56.580791 8756 raft_consensus.cc:729] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 459f935f051e42d4938e20b9e58425b2, State: Initialized, Role: FOLLOWER
I20260806 12:23:56.580806 8753 raft_consensus.cc:729] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b89e5769ac4aeabaa157a85d382d85, State: Initialized, Role: FOLLOWER
I20260806 12:23:56.581526 8753 consensus_queue.cc:260] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } } peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } }
I20260806 12:23:56.581526 8756 consensus_queue.cc:260] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } } peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } }
I20260806 12:23:56.582496 8754 tablet_bootstrap.cc:492] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416: No bootstrap required, opened a new log
I20260806 12:23:56.582762 8754 ts_tablet_manager.cc:1397] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.005s
I20260806 12:23:56.582963 8753 ts_tablet_manager.cc:1428] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20260806 12:23:56.582963 8756 ts_tablet_manager.cc:1428] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20260806 12:23:56.583262 8270 heartbeater.cc:503] Master 127.28.210.126:37041 was elected leader, sending a full tablet report...
I20260806 12:23:56.584298 8488 heartbeater.cc:503] Master 127.28.210.126:37041 was elected leader, sending a full tablet report...
I20260806 12:23:56.592222 8754 raft_consensus.cc:348] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } } peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } }
I20260806 12:23:56.592553 8754 raft_consensus.cc:374] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:56.592618 8754 raft_consensus.cc:729] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e87c17242b84502bb0db4ad9d7cf416, State: Initialized, Role: FOLLOWER
I20260806 12:23:56.593117 8754 consensus_queue.cc:260] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } } peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } }
I20260806 12:23:56.594136 8701 heartbeater.cc:503] Master 127.28.210.126:37041 was elected leader, sending a full tablet report...
I20260806 12:23:56.594642 8754 ts_tablet_manager.cc:1428] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260806 12:23:56.653648 8764 raft_consensus.cc:482] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:56.653893 8764 raft_consensus.cc:504] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } } peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } }
I20260806 12:23:56.655284 8764 leader_election.cc:288] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55b89e5769ac4aeabaa157a85d382d85 (127.28.210.66:41305), 459f935f051e42d4938e20b9e58425b2 (127.28.210.65:37747)
I20260806 12:23:56.661056 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.655827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:46947 (local address 127.28.210.65:37747)
0806 12:23:56.655964 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.655970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.655985 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:23:56.656039 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.656042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.656124 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.656211 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:56.656907 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.657727 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.659575 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.660263 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.660434 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.660490 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.660813 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.660911 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 12:23:56.661056 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.655697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:44079 (local address 127.28.210.66:41305)
0806 12:23:56.655865 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.655870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.655892 (+ 22us) server_negotiation.cc:407] Connection header received
0806 12:23:56.656032 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.656036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.656114 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.656212 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:56.656907 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.657727 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.659575 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.660263 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.660424 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.660490 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.660813 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.660911 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 12:23:56.661062 8708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.655575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:41305 (local address 127.28.210.67:44079)
0806 12:23:56.655749 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.655768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.655889 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.656416 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.656420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.656451 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:56.656692 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.656700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.657908 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.657912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.659382 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.659396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.660447 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.660452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.660665 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.660684 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.660750 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.660870 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":125,"spinlock_wait_cycles":17536}
I20260806 12:23:56.661062 8709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.655625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:37747 (local address 127.28.210.67:46947)
0806 12:23:56.655813 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.655825 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.655889 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.656416 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.656420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.656453 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:56.656665 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.656676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.657908 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.657912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.659382 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.659396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.660466 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.660470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.660611 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.660657 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.660750 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.660870 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":194,"spinlock_wait_cycles":11776}
I20260806 12:23:56.661960 8762 raft_consensus.cc:482] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:56.662220 8762 raft_consensus.cc:504] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } } peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } }
I20260806 12:23:56.662302 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde8d06744444610adbc8636194f9be0" candidate_uuid: "2e87c17242b84502bb0db4ad9d7cf416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "459f935f051e42d4938e20b9e58425b2" is_pre_election: true
I20260806 12:23:56.662631 8191 raft_consensus.cc:2454] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e87c17242b84502bb0db4ad9d7cf416 in term 0.
I20260806 12:23:56.663249 8527 leader_election.cc:302] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e87c17242b84502bb0db4ad9d7cf416, 459f935f051e42d4938e20b9e58425b2; no voters:
I20260806 12:23:56.663559 8764 raft_consensus.cc:2781] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:23:56.663656 8764 raft_consensus.cc:482] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:56.663707 8764 raft_consensus.cc:3037] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:56.663849 8762 leader_election.cc:288] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e87c17242b84502bb0db4ad9d7cf416 (127.28.210.67:33181), 459f935f051e42d4938e20b9e58425b2 (127.28.210.65:37747)
I20260806 12:23:56.664677 8412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde8d06744444610adbc8636194f9be0" candidate_uuid: "2e87c17242b84502bb0db4ad9d7cf416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55b89e5769ac4aeabaa157a85d382d85" is_pre_election: true
I20260806 12:23:56.664944 8412 raft_consensus.cc:2454] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e87c17242b84502bb0db4ad9d7cf416 in term 0.
I20260806 12:23:56.667014 8764 raft_consensus.cc:504] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } } peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } }
I20260806 12:23:56.667869 8412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde8d06744444610adbc8636194f9be0" candidate_uuid: "2e87c17242b84502bb0db4ad9d7cf416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55b89e5769ac4aeabaa157a85d382d85"
I20260806 12:23:56.668030 8412 raft_consensus.cc:3037] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:56.669798 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.664236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:33181 (local address 127.28.210.66:59147)
0806 12:23:56.664609 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.664625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.664716 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.664998 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.665002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.665025 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:56.665299 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.665311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.666419 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.666423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.667812 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.667823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.669203 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.669207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.669352 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.669385 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.669475 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.669572 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300}
I20260806 12:23:56.670019 8412 raft_consensus.cc:2454] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e87c17242b84502bb0db4ad9d7cf416 in term 1.
I20260806 12:23:56.670447 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.664126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:37747 (local address 127.28.210.66:46557)
0806 12:23:56.664414 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:56.664431 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:56.664529 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:56.665951 (+ 1422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:56.665955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:56.665979 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:56.666152 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.666160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.667498 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.667502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.668786 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:56.668801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.669860 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.669864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:56.670016 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.670047 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 12:23:56.670110 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 12:23:56.670288 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260806 12:23:56.670674 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.664377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:59147 (local address 127.28.210.67:33181)
0806 12:23:56.664498 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.664502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.664789 (+ 287us) server_negotiation.cc:407] Connection header received
0806 12:23:56.664841 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.664843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.664887 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.664956 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:56.665443 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.666231 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.668071 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.668694 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.668864 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.668930 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.670338 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.670442 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 12:23:56.670658 8526 leader_election.cc:302] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e87c17242b84502bb0db4ad9d7cf416, 55b89e5769ac4aeabaa157a85d382d85; no voters:
I20260806 12:23:56.671499 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.665149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:46557 (local address 127.28.210.65:37747)
0806 12:23:56.665282 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.665286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.665298 (+ 12us) server_negotiation.cc:407] Connection header received
0806 12:23:56.665345 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.665348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.665390 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.665458 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:56.666347 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.667305 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.669086 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.669677 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.671107 (+ 1430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.671182 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.671235 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.671348 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 12:23:56.672186 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde8d06744444610adbc8636194f9be0" candidate_uuid: "55b89e5769ac4aeabaa157a85d382d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "459f935f051e42d4938e20b9e58425b2" is_pre_election: true
I20260806 12:23:56.672381 8191 raft_consensus.cc:2454] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55b89e5769ac4aeabaa157a85d382d85 in term 0.
I20260806 12:23:56.672456 8638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde8d06744444610adbc8636194f9be0" candidate_uuid: "55b89e5769ac4aeabaa157a85d382d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e87c17242b84502bb0db4ad9d7cf416" is_pre_election: true
I20260806 12:23:56.672767 8638 raft_consensus.cc:2379] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 55b89e5769ac4aeabaa157a85d382d85 in current term 1: Already voted for candidate 2e87c17242b84502bb0db4ad9d7cf416 in this term.
I20260806 12:23:56.672960 8300 leader_election.cc:302] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 459f935f051e42d4938e20b9e58425b2, 55b89e5769ac4aeabaa157a85d382d85; no voters:
I20260806 12:23:56.673487 8762 raft_consensus.cc:2741] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260806 12:23:56.673812 8764 leader_election.cc:288] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [CANDIDATE]: Term 1 election: Requested vote from peers 55b89e5769ac4aeabaa157a85d382d85 (127.28.210.66:41305), 459f935f051e42d4938e20b9e58425b2 (127.28.210.65:37747)
I20260806 12:23:56.673874 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde8d06744444610adbc8636194f9be0" candidate_uuid: "2e87c17242b84502bb0db4ad9d7cf416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "459f935f051e42d4938e20b9e58425b2"
I20260806 12:23:56.673938 8764 raft_consensus.cc:2781] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:23:56.674010 8764 raft_consensus.cc:686] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 1 LEADER]: Becoming Leader. State: Replica: 2e87c17242b84502bb0db4ad9d7cf416, State: Running, Role: LEADER
I20260806 12:23:56.674008 8191 raft_consensus.cc:3037] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:23:56.674361 8764 consensus_queue.cc:237] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } } peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } }
I20260806 12:23:56.676047 8191 raft_consensus.cc:2454] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e87c17242b84502bb0db4ad9d7cf416 in term 1.
I20260806 12:23:56.679075 6793 catalog_manager.cc:5495] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e87c17242b84502bb0db4ad9d7cf416 (127.28.210.67). New cstate: current_term: 1 leader_uuid: "2e87c17242b84502bb0db4ad9d7cf416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } health_report { overall_health: UNKNOWN } } }
I20260806 12:23:56.714166 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.709767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.28.210.67:33181)
0806 12:23:56.709913 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.709917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.709932 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:23:56.710021 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.710023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.710082 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.710179 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:23:56.710437 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.711001 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.712003 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.712373 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.712470 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.712902 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:23:56.713507 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:23:56.713637 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.713764 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.713842 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 12:23:56.715924 8272 tablet.cc:2369] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:23:56.722645 8491 tablet.cc:2369] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:23:56.724846 8191 raft_consensus.cc:1261] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2 [term 1 FOLLOWER]: Refusing update from remote peer 2e87c17242b84502bb0db4ad9d7cf416: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:56.724846 8412 raft_consensus.cc:1261] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [term 1 FOLLOWER]: Refusing update from remote peer 2e87c17242b84502bb0db4ad9d7cf416: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:23:56.725814 8764 consensus_queue.cc:1035] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:56.726136 8765 consensus_queue.cc:1035] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:23:56.737214 8773 mvcc.cc:205] Tried to move back new op lower bound from 7315533974414708736 to 7315533974222905344. Current Snapshot: MvccSnapshot[applied={T|T < 7315533974414708736}]
I20260806 12:23:56.751183 6794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:56.751457 6794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:23:56.757736 6794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82a2d105a1274cba837b796cd94f697c: Sending DeleteTablet for 3 replicas of tablet bde8d06744444610adbc8636194f9be0
I20260806 12:23:56.758586 8378 tablet_service.cc:1514] Processing DeleteTablet for tablet bde8d06744444610adbc8636194f9be0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:56 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:45652
I20260806 12:23:56.759228 8605 tablet_service.cc:1514] Processing DeleteTablet for tablet bde8d06744444610adbc8636194f9be0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:56 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:57510
I20260806 12:23:56.759415 8784 tablet_replica.cc:330] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85: stopping tablet replica
I20260806 12:23:56.759886 8784 raft_consensus.cc:2229] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:56.760171 8785 tablet_replica.cc:330] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416: stopping tablet replica
I20260806 12:23:56.760349 8784 raft_consensus.cc:2258] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:56.760550 8161 tablet_service.cc:1514] Processing DeleteTablet for tablet bde8d06744444610adbc8636194f9be0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:23:56 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:60204
I20260806 12:23:56.760787 8785 raft_consensus.cc:2229] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:23:56.761410 8785 raft_consensus.cc:2258] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:56.762881 8784 ts_tablet_manager.cc:1905] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:56.763641 8785 ts_tablet_manager.cc:1905] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:23:56.764737 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 6744
I20260806 12:23:56.776829 8784 ts_tablet_manager.cc:1918] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:23:56.777020 8784 log.cc:1196] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/bde8d06744444610adbc8636194f9be0
I20260806 12:23:56.777534 8784 ts_tablet_manager.cc:1939] T bde8d06744444610adbc8636194f9be0 P 55b89e5769ac4aeabaa157a85d382d85: Deleting consensus metadata
I20260806 12:23:56.785317 8790 tablet_replica.cc:330] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2: stopping tablet replica
I20260806 12:23:56.785912 8790 raft_consensus.cc:2229] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:23:56.786403 8790 raft_consensus.cc:2258] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:23:56.789348 8785 ts_tablet_manager.cc:1918] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:23:56.789494 8785 log.cc:1196] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/bde8d06744444610adbc8636194f9be0
I20260806 12:23:56.789866 8785 ts_tablet_manager.cc:1939] T bde8d06744444610adbc8636194f9be0 P 2e87c17242b84502bb0db4ad9d7cf416: Deleting consensus metadata
I20260806 12:23:56.792070 8790 ts_tablet_manager.cc:1905] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 12:23:56.795706 8301 connection.cc:535] server connection from 127.0.0.1:45652 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 12:23:56.796888 8526 connection.cc:441] server connection from 127.0.0.1:57510 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57510 (request call id 1) could send its response
I20260806 12:23:56.797811 8790 ts_tablet_manager.cc:1918] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:23:56.798076 8790 log.cc:1196] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-0/wal/wals/bde8d06744444610adbc8636194f9be0
I20260806 12:23:56.798185 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.762791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.28.210.125:46317)
0806 12:23:56.762942 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.762949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.784732 (+ 21783us) server_negotiation.cc:407] Connection header received
0806 12:23:56.785003 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.785008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.785104 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.785243 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.787194 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.788157 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.789700 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.790237 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.790355 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.793016 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.793022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.793026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.793073 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.795255 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.796121 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.796126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.796128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.796255 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.796654 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.796658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.796661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.797082 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.797234 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.797757 (+ 523us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.797971 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:23:56.798796 8790 ts_tablet_manager.cc:1939] T bde8d06744444610adbc8636194f9be0 P 459f935f051e42d4938e20b9e58425b2: Deleting consensus metadata
W20260806 12:23:56.800503 8093 connection.cc:441] server connection from 127.0.0.1:60204 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60204 (request call id 1) could send its response
I20260806 12:23:56.813302 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.763343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.28.210.124:39995)
0806 12:23:56.763489 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.763494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.799143 (+ 35649us) server_negotiation.cc:407] Connection header received
0806 12:23:56.799370 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.799374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.799436 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.799501 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.801729 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.802851 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.804583 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.805111 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.805247 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.808871 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.808876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.808878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.808917 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.810672 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.811474 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.811479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.811482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.811598 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.811999 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.812003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.812005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.812486 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.812633 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.812968 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.813116 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 12:23:56.815151 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.802466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.28.210.125:46317)
0806 12:23:56.802617 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.802622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.803614 (+ 992us) server_negotiation.cc:407] Connection header received
0806 12:23:56.803758 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.803763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.803825 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.803916 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.805819 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.806380 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.807653 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.807890 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.808009 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.809839 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.809844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.809847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.809882 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.811687 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.812227 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.812231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.812233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.812346 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.813575 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.813579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.813582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.813877 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.814032 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.814792 (+ 760us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.814933 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:23:56.817408 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:56.802985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.28.210.124:39995)
0806 12:23:56.803131 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:56.803136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:56.804871 (+ 1735us) server_negotiation.cc:407] Connection header received
0806 12:23:56.805122 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:56.805126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:56.805187 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:56.805247 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:23:56.806840 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.807653 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.809404 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:56.809877 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:56.810004 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:56.812550 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:23:56.812555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:23:56.812558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:23:56.812590 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:23:56.814343 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.815758 (+ 1415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.815763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.815765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.815916 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:23:56.816352 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:23:56.816356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:23:56.816358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:23:56.816614 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:23:56.816787 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:56.817063 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 12:23:56.817216 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 12:23:57.741370 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:57.740813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.65:34343)
0806 12:23:57.741156 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:57.741257 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":90,"threads_started":1}
W20260806 12:23:57.741750 8270 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:37041 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
I20260806 12:23:57.742861 8805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:57.742389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.67:38473)
0806 12:23:57.742657 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:57.742753 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260806 12:23:57.743117 8806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:57.742624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.66:55875)
0806 12:23:57.742896 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:57.742992 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
W20260806 12:23:57.743371 8701 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:37041 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
W20260806 12:23:57.743588 8488 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:37041 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
I20260806 12:23:58.257835 8808 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82a2d105a1274cba837b796cd94f697c)
I20260806 12:23:58.257815 8809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82a2d105a1274cba837b796cd94f697c)
I20260806 12:23:58.258122 8808 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } }
I20260806 12:23:58.258196 8809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } }
I20260806 12:23:58.259316 8808 leader_election.cc:288] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041), 3c7a6f38f9fc40f187bf1ef7334aca6a (127.28.210.125:46317)
I20260806 12:23:58.259565 8809 leader_election.cc:288] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041), c0c3343c36004df38a7b11f49bf19adc (127.28.210.124:39995)
I20260806 12:23:58.260152 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.259531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:38318)
0806 12:23:58.259924 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.260008 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":77,"threads_started":1}
I20260806 12:23:58.260154 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.259531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:38320)
0806 12:23:58.259907 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.260005 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":84,"threads_started":1}
W20260806 12:23:58.260351 7699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
W20260806 12:23:58.260351 7975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
I20260806 12:23:58.261540 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.261247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:38350)
0806 12:23:58.261397 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.261456 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 12:23:58.261682 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.261247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:38338)
0806 12:23:58.261549 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.261608 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
W20260806 12:23:58.261710 7975 leader_election.cc:334] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041): Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
W20260806 12:23:58.261802 7699 leader_election.cc:334] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041): Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
I20260806 12:23:58.265781 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.259531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:39995 (local address 127.0.0.1:45290)
0806 12:23:58.260537 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.260553 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:58.260733 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:58.261452 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:58.261455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:58.261475 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:58.261580 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:58.261587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:58.262499 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:58.262503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:58.263953 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:58.263970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:58.265142 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:58.265146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:58.265338 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:58.265370 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 12:23:58.265481 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 12:23:58.265601 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":261,"thread_start_us":55,"threads_started":1}
I20260806 12:23:58.265827 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.259924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:46317 (local address 127.0.0.1:43586)
0806 12:23:58.260616 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.260634 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:58.260762 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:58.261273 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:58.261277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:58.261301 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:58.261422 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:58.261430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:58.262544 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:58.262548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:58.263953 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:58.263970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:58.265141 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:58.265146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:58.265476 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:58.265503 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 12:23:58.265595 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 12:23:58.265691 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":138,"thread_start_us":72,"threads_started":1}
I20260806 12:23:58.265857 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.260160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45290 (local address 127.28.210.124:39995)
0806 12:23:58.260387 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:58.260393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:58.260632 (+ 239us) server_negotiation.cc:407] Connection header received
0806 12:23:58.260911 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:58.260914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:58.260957 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:58.261381 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:58.261721 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:58.262347 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:58.264227 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:58.264947 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:58.265112 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:58.265379 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:58.265534 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 12:23:58.265659 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20260806 12:23:58.266181 8816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.259949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.28.210.125:46317)
0806 12:23:58.260553 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:58.260558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:58.260861 (+ 303us) server_negotiation.cc:407] Connection header received
0806 12:23:58.260909 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:58.260912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:58.260957 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:58.261042 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:58.261549 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:58.262424 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:58.264227 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:58.264947 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:58.265112 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:58.265360 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:58.265640 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 12:23:58.265727 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":220,"thread_start_us":55,"threads_started":1}
I20260806 12:23:58.266535 8006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c0c3343c36004df38a7b11f49bf19adc" is_pre_election: true
I20260806 12:23:58.266649 7730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0c3343c36004df38a7b11f49bf19adc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" is_pre_election: true
I20260806 12:23:58.266834 7730 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0c3343c36004df38a7b11f49bf19adc in term 1.
I20260806 12:23:58.266834 8006 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c7a6f38f9fc40f187bf1ef7334aca6a in term 1.
I20260806 12:23:58.267278 7697 leader_election.cc:302] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c7a6f38f9fc40f187bf1ef7334aca6a, c0c3343c36004df38a7b11f49bf19adc; no voters: 82a2d105a1274cba837b796cd94f697c
I20260806 12:23:58.267302 7973 leader_election.cc:302] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c7a6f38f9fc40f187bf1ef7334aca6a, c0c3343c36004df38a7b11f49bf19adc; no voters: 82a2d105a1274cba837b796cd94f697c
I20260806 12:23:58.267541 8808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 12:23:58.267649 8808 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82a2d105a1274cba837b796cd94f697c)
I20260806 12:23:58.267711 8808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 1 FOLLOWER]: Advancing to term 2
I20260806 12:23:58.267755 8809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 12:23:58.267815 8809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82a2d105a1274cba837b796cd94f697c)
I20260806 12:23:58.267863 8809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 1 FOLLOWER]: Advancing to term 2
I20260806 12:23:58.269260 8808 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } }
I20260806 12:23:58.269260 8809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } }
I20260806 12:23:58.269790 8809 leader_election.cc:288] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [CANDIDATE]: Term 2 election: Requested vote from peers 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041), c0c3343c36004df38a7b11f49bf19adc (127.28.210.124:39995)
I20260806 12:23:58.269791 8808 leader_election.cc:288] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [CANDIDATE]: Term 2 election: Requested vote from peers 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041), 3c7a6f38f9fc40f187bf1ef7334aca6a (127.28.210.125:46317)
I20260806 12:23:58.270388 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.269996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:38356)
0806 12:23:58.270186 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.270255 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 12:23:58.270350 8006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c0c3343c36004df38a7b11f49bf19adc"
I20260806 12:23:58.270346 7730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0c3343c36004df38a7b11f49bf19adc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a"
I20260806 12:23:58.270447 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.270107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:38358)
0806 12:23:58.270313 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.270372 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260806 12:23:58.270537 7730 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c0c3343c36004df38a7b11f49bf19adc in current term 2: Already voted for candidate 3c7a6f38f9fc40f187bf1ef7334aca6a in this term.
I20260806 12:23:58.270540 8006 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c7a6f38f9fc40f187bf1ef7334aca6a in current term 2: Already voted for candidate c0c3343c36004df38a7b11f49bf19adc in this term.
I20260806 12:23:58.271204 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.270928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:38370)
0806 12:23:58.271067 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.271129 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 12:23:58.271354 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.270948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:38382)
0806 12:23:58.271229 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.271295 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
W20260806 12:23:58.271334 7699 leader_election.cc:334] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041): Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
I20260806 12:23:58.271463 7699 leader_election.cc:302] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c7a6f38f9fc40f187bf1ef7334aca6a; no voters: 82a2d105a1274cba837b796cd94f697c, c0c3343c36004df38a7b11f49bf19adc
W20260806 12:23:58.271464 7975 leader_election.cc:334] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041): Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
I20260806 12:23:58.271533 7975 leader_election.cc:302] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0c3343c36004df38a7b11f49bf19adc; no voters: 3c7a6f38f9fc40f187bf1ef7334aca6a, 82a2d105a1274cba837b796cd94f697c
I20260806 12:23:58.271634 8809 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 12:23:58.271673 8808 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 12:23:58.743099 8821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.742586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.65:38555)
0806 12:23:58.742916 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.742991 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":103,"threads_started":1}
I20260806 12:23:58.744669 8822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.744065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.67:34205)
0806 12:23:58.744500 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.744570 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":103,"threads_started":1}
I20260806 12:23:58.744669 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:58.744199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.66:58365)
0806 12:23:58.744492 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:58.744563 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":65,"threads_started":1}
I20260806 12:23:59.744632 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.744118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.65:53875)
0806 12:23:59.744455 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.744523 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
I20260806 12:23:59.746068 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.745611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.66:57079)
0806 12:23:59.745875 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.745963 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260806 12:23:59.746068 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.745708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.67:50735)
0806 12:23:59.745928 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.745982 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
I20260806 12:23:59.872113 8828 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:23:59.872267 8828 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } }
I20260806 12:23:59.872725 8828 leader_election.cc:288] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041), 3c7a6f38f9fc40f187bf1ef7334aca6a (127.28.210.125:46317)
I20260806 12:23:59.873384 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.872951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:54476)
0806 12:23:59.873188 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.873267 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260806 12:23:59.873344 7730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0c3343c36004df38a7b11f49bf19adc" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" is_pre_election: true
I20260806 12:23:59.873523 7730 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0c3343c36004df38a7b11f49bf19adc in term 2.
I20260806 12:23:59.873888 7973 leader_election.cc:302] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c7a6f38f9fc40f187bf1ef7334aca6a, c0c3343c36004df38a7b11f49bf19adc; no voters:
I20260806 12:23:59.874061 8828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 12:23:59.874110 8828 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:23:59.874147 8828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 2 FOLLOWER]: Advancing to term 3
I20260806 12:23:59.874261 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.874030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:54480)
0806 12:23:59.874130 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.874190 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260806 12:23:59.874374 7975 leader_election.cc:334] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041): Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
I20260806 12:23:59.875267 8828 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } }
I20260806 12:23:59.875640 8828 leader_election.cc:288] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [CANDIDATE]: Term 3 election: Requested vote from peers 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041), 3c7a6f38f9fc40f187bf1ef7334aca6a (127.28.210.125:46317)
I20260806 12:23:59.876117 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.875785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:54490)
0806 12:23:59.875945 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.876010 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 12:23:59.876057 7730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0c3343c36004df38a7b11f49bf19adc" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a"
I20260806 12:23:59.876235 7730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 2 FOLLOWER]: Advancing to term 3
I20260806 12:23:59.876876 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.876637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:54500)
0806 12:23:59.876728 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.876786 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260806 12:23:59.877009 7975 leader_election.cc:334] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041): Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
I20260806 12:23:59.877552 7730 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0c3343c36004df38a7b11f49bf19adc in term 3.
I20260806 12:23:59.877885 7973 leader_election.cc:302] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c7a6f38f9fc40f187bf1ef7334aca6a, c0c3343c36004df38a7b11f49bf19adc; no voters: 82a2d105a1274cba837b796cd94f697c
I20260806 12:23:59.878057 8828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 3 FOLLOWER]: Leader election won for term 3
I20260806 12:23:59.878291 8828 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [term 3 LEADER]: Becoming Leader. State: Replica: c0c3343c36004df38a7b11f49bf19adc, State: Running, Role: LEADER
I20260806 12:23:59.878568 8828 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } }
I20260806 12:23:59.879768 8832 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0c3343c36004df38a7b11f49bf19adc. Latest consensus state: current_term: 3 leader_uuid: "c0c3343c36004df38a7b11f49bf19adc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } } }
I20260806 12:23:59.879877 8832 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:59.880164 8834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:23:59.882624 8834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:23:59.883034 8834 catalog_manager.cc:1259] Loaded cluster ID: 8165e75ab35f4f5aadc5653b708b0362
I20260806 12:23:59.883081 8834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:23:59.883633 8834 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:23:59.884040 8834 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: Loaded TSK: 0
I20260806 12:23:59.884781 8834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:23:59.893953 7989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:23:59.897610 7989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:23:59.901213 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.900916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:54510)
0806 12:23:59.901044 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.901117 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 12:23:59.901330 7730 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [term 3 FOLLOWER]: Refusing update from remote peer c0c3343c36004df38a7b11f49bf19adc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260806 12:23:59.901865 8832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 12:23:59.901980 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.901769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.0.0.1:54512)
0806 12:23:59.901875 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.901924 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 12:23:59.902225 7975 consensus_peers.cc:487] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc -> Peer 82a2d105a1274cba837b796cd94f697c (127.28.210.126:37041): Couldn't send request to peer 82a2d105a1274cba837b796cd94f697c. Status: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 12:23:59.903407 8836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0c3343c36004df38a7b11f49bf19adc. Latest consensus state: current_term: 3 leader_uuid: "c0c3343c36004df38a7b11f49bf19adc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } } }
I20260806 12:23:59.903563 8836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:23:59.904783 8828 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c0c3343c36004df38a7b11f49bf19adc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } } }
I20260806 12:23:59.904910 8828 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc [sys.catalog]: This master's current role is: LEADER
I20260806 12:23:59.905169 8836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c0c3343c36004df38a7b11f49bf19adc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "82a2d105a1274cba837b796cd94f697c" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 37041 } } peers { permanent_uuid: "3c7a6f38f9fc40f187bf1ef7334aca6a" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 46317 } attrs { promote: false } } peers { permanent_uuid: "c0c3343c36004df38a7b11f49bf19adc" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 39995 } attrs { promote: false } } }
I20260806 12:23:59.905582 8836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7a6f38f9fc40f187bf1ef7334aca6a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:23:59.916546 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.912861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.28.210.66:41305)
0806 12:23:59.913104 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:59.913108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:59.913121 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:23:59.913170 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:59.913172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:59.913215 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:59.913299 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:59.913640 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:59.914251 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:59.915397 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:59.915979 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:59.916111 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:59.916186 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:59.916326 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 12:23:59.916408 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260806 12:23:59.916554 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.911947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:41305 (local address 127.0.0.1:39810)
0806 12:23:59.912937 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.912952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:59.913051 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:59.913405 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:59.913408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:59.913425 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:59.913511 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:59.913516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:59.914357 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:59.914361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:59.915249 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:59.915261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:59.916120 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:59.916123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:59.916206 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:59.916231 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 12:23:59.916303 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 12:23:59.916383 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":103,"threads_started":1}
I20260806 12:23:59.916769 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.911655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:37747 (local address 127.0.0.1:40030)
0806 12:23:59.911822 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.911838 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:59.911936 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:59.912661 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:59.912665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:59.912688 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:59.912811 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:59.912817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:59.913928 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:59.913932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:59.915194 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:59.915205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:59.916410 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:59.916414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:59.916498 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:59.916524 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 12:23:59.916568 (+ 44us) client_negotiation.cc:240] Negotiation successful
0806 12:23:59.916638 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":19}
I20260806 12:23:59.917085 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.912339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:33181 (local address 127.0.0.1:51124)
0806 12:23:59.913175 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.913188 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:23:59.913275 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:23:59.913504 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:23:59.913507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:23:59.913523 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:23:59.913645 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:59.913650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:59.914602 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:59.914605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:59.915738 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:23:59.915748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:23:59.916689 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:23:59.916692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:23:59.916785 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:59.916819 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 12:23:59.916891 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 12:23:59.916956 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":22,"thread_start_us":317,"threads_started":1}
I20260806 12:23:59.917147 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.912440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.28.210.67:33181)
0806 12:23:59.912919 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:59.912923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:59.913264 (+ 341us) server_negotiation.cc:407] Connection header received
0806 12:23:59.913367 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:59.913370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:59.913413 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:59.913488 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:59.913749 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:59.914472 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:59.915884 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:59.916476 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:59.916550 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:59.916607 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:59.916928 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 12:23:59.917015 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":84,"threads_started":1}
I20260806 12:23:59.917184 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.911748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.28.210.65:37747)
0806 12:23:59.912283 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0806 12:23:59.912288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:23:59.912303 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:23:59.912360 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:23:59.912363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:23:59.912414 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:23:59.912926 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:23:59.913052 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:59.913813 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:59.915346 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:23:59.916000 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:23:59.916882 (+ 882us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:23:59.916951 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 12:23:59.916988 (+ 37us) server_negotiation.cc:299] Negotiation successful
0806 12:23:59.917065 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":105,"threads_started":1}
I20260806 12:23:59.917703 8605 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab10ba8c0064aeeb334d1c020ab548a (DEFAULT_TABLE table=test-table [id=c9507bd3d7ee4a8889bc3a07f5f1cce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:59.917853 8161 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab10ba8c0064aeeb334d1c020ab548a (DEFAULT_TABLE table=test-table [id=c9507bd3d7ee4a8889bc3a07f5f1cce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:59.918036 8605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab10ba8c0064aeeb334d1c020ab548a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:59.918032 8378 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab10ba8c0064aeeb334d1c020ab548a (DEFAULT_TABLE table=test-table [id=c9507bd3d7ee4a8889bc3a07f5f1cce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:23:59.918107 8161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab10ba8c0064aeeb334d1c020ab548a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:59.918275 8378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab10ba8c0064aeeb334d1c020ab548a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:23:59.920275 8851 tablet_bootstrap.cc:492] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2: Bootstrap starting.
I20260806 12:23:59.920967 8855 tablet_bootstrap.cc:492] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85: Bootstrap starting.
I20260806 12:23:59.921591 8851 tablet_bootstrap.cc:654] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:59.921896 8855 tablet_bootstrap.cc:654] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:59.922575 8852 tablet_bootstrap.cc:492] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416: Bootstrap starting.
I20260806 12:23:59.923039 8851 tablet_bootstrap.cc:492] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2: No bootstrap required, opened a new log
I20260806 12:23:59.923136 8851 ts_tablet_manager.cc:1397] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:23:59.923439 8852 tablet_bootstrap.cc:654] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416: Neither blocks nor log segments found. Creating new log.
I20260806 12:23:59.923604 8851 raft_consensus.cc:348] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } } peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } }
I20260806 12:23:59.923719 8851 raft_consensus.cc:374] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:59.923751 8851 raft_consensus.cc:729] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 459f935f051e42d4938e20b9e58425b2, State: Initialized, Role: FOLLOWER
I20260806 12:23:59.923884 8851 consensus_queue.cc:260] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } } peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } }
I20260806 12:23:59.924306 8851 ts_tablet_manager.cc:1428] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260806 12:23:59.924508 8855 tablet_bootstrap.cc:492] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85: No bootstrap required, opened a new log
I20260806 12:23:59.924615 8855 ts_tablet_manager.cc:1397] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 12:23:59.925109 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.924818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.65:33573)
0806 12:23:59.924949 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.925020 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 12:23:59.925961 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.925649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.65:44261)
0806 12:23:59.925802 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.925870 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 12:23:59.926719 8855 raft_consensus.cc:348] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } } peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } }
I20260806 12:23:59.926891 8855 raft_consensus.cc:374] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:59.926932 8855 raft_consensus.cc:729] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b89e5769ac4aeabaa157a85d382d85, State: Initialized, Role: FOLLOWER
I20260806 12:23:59.927114 8268 heartbeater.cc:503] Master 127.28.210.124:39995 was elected leader, sending a full tablet report...
I20260806 12:23:59.927807 8852 tablet_bootstrap.cc:492] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416: No bootstrap required, opened a new log
I20260806 12:23:59.927932 8852 ts_tablet_manager.cc:1397] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 12:23:59.928213 8855 consensus_queue.cc:260] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } } peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } }
I20260806 12:23:59.928737 8855 ts_tablet_manager.cc:1428] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 12:23:59.929347 8852 raft_consensus.cc:348] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } } peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } }
I20260806 12:23:59.929535 8852 raft_consensus.cc:374] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:23:59.929587 8852 raft_consensus.cc:729] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e87c17242b84502bb0db4ad9d7cf416, State: Initialized, Role: FOLLOWER
I20260806 12:23:59.929698 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.929329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.66:56939)
0806 12:23:59.929535 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.929608 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260806 12:23:59.929749 8852 consensus_queue.cc:260] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } } peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } }
I20260806 12:23:59.930260 8852 ts_tablet_manager.cc:1428] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 12:23:59.930457 8486 heartbeater.cc:503] Master 127.28.210.124:39995 was elected leader, sending a full tablet report...
I20260806 12:23:59.930614 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.930263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.66:51265)
0806 12:23:59.930462 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.930530 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260806 12:23:59.931841 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.931459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.67:41047)
0806 12:23:59.931664 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.931738 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 12:23:59.933571 8700 heartbeater.cc:503] Master 127.28.210.124:39995 was elected leader, sending a full tablet report...
I20260806 12:23:59.934558 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:23:59.934202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.67:55951)
0806 12:23:59.934376 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 12:23:59.934454 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260806 12:23:59.991047 8703 tablet.cc:2369] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:24:00.011762 8860 raft_consensus.cc:482] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:24:00.011969 8860 raft_consensus.cc:504] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } } peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } }
I20260806 12:24:00.012699 8860 leader_election.cc:288] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 459f935f051e42d4938e20b9e58425b2 (127.28.210.65:37747), 2e87c17242b84502bb0db4ad9d7cf416 (127.28.210.67:33181)
I20260806 12:24:00.013211 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab10ba8c0064aeeb334d1c020ab548a" candidate_uuid: "55b89e5769ac4aeabaa157a85d382d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "459f935f051e42d4938e20b9e58425b2" is_pre_election: true
I20260806 12:24:00.014050 8191 raft_consensus.cc:2454] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55b89e5769ac4aeabaa157a85d382d85 in term 0.
I20260806 12:24:00.014063 8638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab10ba8c0064aeeb334d1c020ab548a" candidate_uuid: "55b89e5769ac4aeabaa157a85d382d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e87c17242b84502bb0db4ad9d7cf416" is_pre_election: true
I20260806 12:24:00.014318 8638 raft_consensus.cc:2454] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55b89e5769ac4aeabaa157a85d382d85 in term 0.
I20260806 12:24:00.014699 8300 leader_election.cc:302] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e87c17242b84502bb0db4ad9d7cf416, 459f935f051e42d4938e20b9e58425b2, 55b89e5769ac4aeabaa157a85d382d85; no voters:
I20260806 12:24:00.014971 8860 raft_consensus.cc:2781] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:24:00.015062 8860 raft_consensus.cc:482] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:24:00.015110 8860 raft_consensus.cc:3037] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:00.016785 8860 raft_consensus.cc:504] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } } peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } }
I20260806 12:24:00.017329 8860 leader_election.cc:288] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [CANDIDATE]: Term 1 election: Requested vote from peers 459f935f051e42d4938e20b9e58425b2 (127.28.210.65:37747), 2e87c17242b84502bb0db4ad9d7cf416 (127.28.210.67:33181)
I20260806 12:24:00.017839 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab10ba8c0064aeeb334d1c020ab548a" candidate_uuid: "55b89e5769ac4aeabaa157a85d382d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "459f935f051e42d4938e20b9e58425b2"
I20260806 12:24:00.017839 8638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab10ba8c0064aeeb334d1c020ab548a" candidate_uuid: "55b89e5769ac4aeabaa157a85d382d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e87c17242b84502bb0db4ad9d7cf416"
I20260806 12:24:00.018013 8191 raft_consensus.cc:3037] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:00.018013 8638 raft_consensus.cc:3037] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:00.019277 8191 raft_consensus.cc:2454] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55b89e5769ac4aeabaa157a85d382d85 in term 1.
I20260806 12:24:00.019277 8638 raft_consensus.cc:2454] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55b89e5769ac4aeabaa157a85d382d85 in term 1.
I20260806 12:24:00.019809 8300 leader_election.cc:302] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 459f935f051e42d4938e20b9e58425b2, 55b89e5769ac4aeabaa157a85d382d85; no voters:
I20260806 12:24:00.020105 8860 raft_consensus.cc:2781] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:24:00.020398 8860 raft_consensus.cc:686] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 1 LEADER]: Becoming Leader. State: Replica: 55b89e5769ac4aeabaa157a85d382d85, State: Running, Role: LEADER
I20260806 12:24:00.020735 8860 consensus_queue.cc:237] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } } peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } }
I20260806 12:24:00.021277 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:00.020951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.66:60583)
0806 12:24:00.021145 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:00.021198 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260806 12:24:00.023356 7989 catalog_manager.cc:5495] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55b89e5769ac4aeabaa157a85d382d85 (127.28.210.66). New cstate: current_term: 1 leader_uuid: "55b89e5769ac4aeabaa157a85d382d85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55b89e5769ac4aeabaa157a85d382d85" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 41305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 } health_report { overall_health: UNKNOWN } } }
I20260806 12:24:00.052913 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:00.048549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.28.210.66:41305)
0806 12:24:00.048704 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:00.048709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:00.048725 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:24:00.048819 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:00.048822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:00.048883 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:00.048981 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:24:00.049279 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:00.049878 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:00.050900 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:00.051167 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:00.051281 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:00.051661 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:24:00.052274 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:24:00.052381 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:00.052614 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 12:24:00.052715 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 12:24:00.056937 8191 raft_consensus.cc:1261] T 9ab10ba8c0064aeeb334d1c020ab548a P 459f935f051e42d4938e20b9e58425b2 [term 1 FOLLOWER]: Refusing update from remote peer 55b89e5769ac4aeabaa157a85d382d85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:00.057138 8638 raft_consensus.cc:1261] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416 [term 1 FOLLOWER]: Refusing update from remote peer 55b89e5769ac4aeabaa157a85d382d85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:00.057595 8860 consensus_queue.cc:1035] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "459f935f051e42d4938e20b9e58425b2" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:00.057826 8865 consensus_queue.cc:1035] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e87c17242b84502bb0db4ad9d7cf416" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:00.060563 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:00.060177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.65:52873)
0806 12:24:00.060347 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:00.060449 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 12:24:00.062884 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:00.062518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.67:54615)
0806 12:24:00.062715 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:00.062783 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 12:24:00.064452 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:00.064135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.66:58519)
0806 12:24:00.064301 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:00.064361 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 12:24:00.065209 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:00.064926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.66:56951)
0806 12:24:00.065082 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:00.065136 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 12:24:00.065429 8875 mvcc.cc:205] Tried to move back new op lower bound from 7315533988066156544 to 7315533987927908352. Current Snapshot: MvccSnapshot[applied={T|T < 7315533988066156544}]
I20260806 12:24:00.066059 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:00.065736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.67:48465)
0806 12:24:00.065889 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:00.065961 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 12:24:00.066592 8871 mvcc.cc:205] Tried to move back new op lower bound from 7315533988066156544 to 7315533987927908352. Current Snapshot: MvccSnapshot[applied={T|T < 7315533988066156544}]
I20260806 12:24:00.067400 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:00.066857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:37041 (local address 127.28.210.65:60395)
0806 12:24:00.067246 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:00.067315 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:37041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306}
I20260806 12:24:00.078284 7991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:24:00.078540 7991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:24:00.083606 7991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0c3343c36004df38a7b11f49bf19adc: Sending DeleteTablet for 3 replicas of tablet 9ab10ba8c0064aeeb334d1c020ab548a
I20260806 12:24:00.084544 8161 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab10ba8c0064aeeb334d1c020ab548a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:00 UTC) from {username='oryx', principal='oryx/127.28.210.124@KRBTEST.COM'} at 127.0.0.1:40030
I20260806 12:24:00.084729 8605 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab10ba8c0064aeeb334d1c020ab548a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:00 UTC) from {username='oryx', principal='oryx/127.28.210.124@KRBTEST.COM'} at 127.0.0.1:51124
I20260806 12:24:00.085000 8378 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab10ba8c0064aeeb334d1c020ab548a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:00 UTC) from {username='oryx', principal='oryx/127.28.210.124@KRBTEST.COM'} at 127.0.0.1:39810
I20260806 12:24:00.085361 8891 tablet_replica.cc:330] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416: stopping tablet replica
I20260806 12:24:00.085381 8893 tablet_replica.cc:330] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85: stopping tablet replica
I20260806 12:24:00.085564 8891 raft_consensus.cc:2229] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:24:00.085711 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 7936
I20260806 12:24:00.085530 8893 raft_consensus.cc:2229] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:24:00.085873 8891 raft_consensus.cc:2258] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:00.149204 8891 ts_tablet_manager.cc:1905] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:00.152586 8893 raft_consensus.cc:2258] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:00.153152 8891 ts_tablet_manager.cc:1918] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:24:00.153260 8891 log.cc:1196] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/9ab10ba8c0064aeeb334d1c020ab548a
I20260806 12:24:00.153538 8891 ts_tablet_manager.cc:1939] T 9ab10ba8c0064aeeb334d1c020ab548a P 2e87c17242b84502bb0db4ad9d7cf416: Deleting consensus metadata
I20260806 12:24:00.154253 8893 ts_tablet_manager.cc:1905] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:00.154628 7976 catalog_manager.cc:4841] TS 2e87c17242b84502bb0db4ad9d7cf416 (127.28.210.67:33181): tablet 9ab10ba8c0064aeeb334d1c020ab548a (table test-table [id=c9507bd3d7ee4a8889bc3a07f5f1cce6]) successfully deleted
I20260806 12:24:00.157936 8893 ts_tablet_manager.cc:1918] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:24:00.158078 8893 log.cc:1196] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/9ab10ba8c0064aeeb334d1c020ab548a
I20260806 12:24:00.158424 8893 ts_tablet_manager.cc:1939] T 9ab10ba8c0064aeeb334d1c020ab548a P 55b89e5769ac4aeabaa157a85d382d85: Deleting consensus metadata
I20260806 12:24:00.159812 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 8279
I20260806 12:24:00.159765 7973 catalog_manager.cc:4841] TS 55b89e5769ac4aeabaa157a85d382d85 (127.28.210.66:41305): tablet 9ab10ba8c0064aeeb334d1c020ab548a (table test-table [id=c9507bd3d7ee4a8889bc3a07f5f1cce6]) successfully deleted
I20260806 12:24:00.163647 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:00.159356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:37747 (local address 127.0.0.1:40036)
0806 12:24:00.159685 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:00.159744 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231}
W20260806 12:24:00.164013 7974 catalog_manager.cc:4584] TS 459f935f051e42d4938e20b9e58425b2 (127.28.210.65:37747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ab10ba8c0064aeeb334d1c020ab548a: Network error: Client connection negotiation failed: client connection to 127.28.210.65:37747: connect: Connection refused (error 111)
I20260806 12:24:00.173408 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 8496
I20260806 12:24:00.197106 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 7509
I20260806 12:24:00.217036 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:00.216699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:37747 (local address 127.0.0.1:40046)
0806 12:24:00.216861 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:00.216942 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 12:24:00.217943 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:00.217681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:37747 (local address 127.0.0.1:40052)
0806 12:24:00.217820 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:00.217876 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 12:24:00.219190 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 7672
2026-08-06T12:24:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7746 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:24:00 dist-test-slave-lgr5 krb5kdc[8897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:24:00 dist-test-slave-lgr5 krb5kdc[8897](info): set up 2 sockets
Aug 06 12:24:00 dist-test-slave-lgr5 krb5kdc[8897](info): commencing operation
krb5kdc: starting...
W20260806 12:24:01.435621 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:24:01 dist-test-slave-lgr5 krb5kdc[8897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:24:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:24:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.126.keytab.
Entry for principal oryx/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.126.keytab.
I20260806 12:24:01.576681 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:40067
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.126.keytab
--principal=oryx/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:40067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:01.701483 8926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:01.701843 8926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:01.701905 8926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:01.706249 8926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:24:01.706326 8926 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 12:24:01.706348 8926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:01.706372 8926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:01.706393 8926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:24:01.706414 8926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:24:01.712195 8926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:40067
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.126.keytab
--principal=oryx/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:40067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:01.713433 8926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:01.714560 8926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:01.724435 8935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:01.724520 8926 server_base.cc:1034] running on GCE node
W20260806 12:24:01.724471 8936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:01.724435 8938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:01.725474 8926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:01.726800 8926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:01.727999 8926 hybrid_clock.cc:648] HybridClock initialized: now 1786019041727981 us; error 34 us; skew 500 ppm
Aug 06 12:24:01 dist-test-slave-lgr5 krb5kdc[8897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019041, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:01.731192 8926 init.cc:377] Logged in from keytab as oryx/127.28.210.126@KRBTEST.COM (short username oryx)
I20260806 12:24:01.732275 8926 webserver.cc:460] Webserver started at http://127.28.210.126:34219/ using document root <none> and password file <none>
I20260806 12:24:01.732961 8926 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:01.733039 8926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:01.733263 8926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:01.735560 8926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "28fed4920a9346c6abf99c9e4a50d3fa"
format_stamp: "Formatted at 2026-08-06 12:24:01 on dist-test-slave-lgr5"
server_key: "9afbcdb7c23e71a85c02ee4085273d35"
server_key_iv: "93b5270903be8bfa1acbba28fbce0b3d"
server_key_version: "encryptionkey@0"
I20260806 12:24:01.736199 8926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "28fed4920a9346c6abf99c9e4a50d3fa"
format_stamp: "Formatted at 2026-08-06 12:24:01 on dist-test-slave-lgr5"
server_key: "9afbcdb7c23e71a85c02ee4085273d35"
server_key_iv: "93b5270903be8bfa1acbba28fbce0b3d"
server_key_version: "encryptionkey@0"
I20260806 12:24:01.740381 8926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:24:01.743233 8946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:01.744661 8926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:24:01.744802 8926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "28fed4920a9346c6abf99c9e4a50d3fa"
format_stamp: "Formatted at 2026-08-06 12:24:01 on dist-test-slave-lgr5"
server_key: "9afbcdb7c23e71a85c02ee4085273d35"
server_key_iv: "93b5270903be8bfa1acbba28fbce0b3d"
server_key_version: "encryptionkey@0"
I20260806 12:24:01.744931 8926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:01.758531 8926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:01.761874 8926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:01.762105 8926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:01.770373 8926 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:40067
I20260806 12:24:01.770387 9039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:40067 every 8 connection(s)
I20260806 12:24:01.771402 8926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:24:01.774426 9040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:01.775352 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 8926
I20260806 12:24:01.775475 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:24:01.775761 29513 external_mini_cluster.cc:1428] Setting key b0d1e79de8145b827628c46aaf0d171f
I20260806 12:24:01.781414 9040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa: Bootstrap starting.
I20260806 12:24:01.783959 9040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:01.784811 9040 log.cc:824] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa: Log is configured to *not* fsync() on all Append() calls
Aug 06 12:24:01 dist-test-slave-lgr5 krb5kdc[8897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:24:01.787173 9040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa: No bootstrap required, opened a new log
I20260806 12:24:01.790822 9040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fed4920a9346c6abf99c9e4a50d3fa" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40067 } }
I20260806 12:24:01.791071 9040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:01.791108 9040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fed4920a9346c6abf99c9e4a50d3fa, State: Initialized, Role: FOLLOWER
I20260806 12:24:01.791229 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:01.777540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.28.210.126:40067)
0806 12:24:01.778052 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:01.778069 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:01.778109 (+ 40us) server_negotiation.cc:407] Connection header received
0806 12:24:01.778786 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:01.778805 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:01.779298 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:01.779666 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:01.780560 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:01.781612 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:01.782456 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:01.782937 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:01.783089 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:01.785812 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:01.785818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:01.785835 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:01.785863 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:01.788296 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:01.788795 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:01.788799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:01.788804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:01.788913 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:01.789197 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:01.789199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:01.789200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:01.789616 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:01.789786 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:01.790056 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 12:24:01.790364 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":162,"threads_started":1}
I20260806 12:24:01.791664 9040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fed4920a9346c6abf99c9e4a50d3fa" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40067 } }
I20260806 12:24:01.791801 9040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:24:01.791863 9040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:24:01.791957 9040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:01.792950 9040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fed4920a9346c6abf99c9e4a50d3fa" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40067 } }
I20260806 12:24:01.793291 9040 leader_election.cc:302] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28fed4920a9346c6abf99c9e4a50d3fa; no voters:
I20260806 12:24:01.793609 9040 leader_election.cc:288] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:24:01.793704 9046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:24:01.793931 9046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [term 1 LEADER]: Becoming Leader. State: Replica: 28fed4920a9346c6abf99c9e4a50d3fa, State: Running, Role: LEADER
I20260806 12:24:01.794241 9046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fed4920a9346c6abf99c9e4a50d3fa" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40067 } }
I20260806 12:24:01.794806 9040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:24:01.795912 9050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28fed4920a9346c6abf99c9e4a50d3fa. Latest consensus state: current_term: 1 leader_uuid: "28fed4920a9346c6abf99c9e4a50d3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fed4920a9346c6abf99c9e4a50d3fa" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40067 } } }
I20260806 12:24:01.796044 9050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [sys.catalog]: This master's current role is: LEADER
I20260806 12:24:01.796180 9049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28fed4920a9346c6abf99c9e4a50d3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fed4920a9346c6abf99c9e4a50d3fa" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40067 } } }
I20260806 12:24:01.796305 9049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa [sys.catalog]: This master's current role is: LEADER
I20260806 12:24:01.796645 9063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:24:01.799894 9063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:24:01.805217 9063 catalog_manager.cc:1347] Generated new cluster ID: ddc60b89faeb4ed1ae842923e85cf779
I20260806 12:24:01.805307 9063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:24:01.821805 9063 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:24:01.822892 9063 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:24:01.827417 9063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa: Generated new TSK 0
I20260806 12:24:01.828286 9063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab.
Entry for principal oryx/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab.
I20260806 12:24:01.872974 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:40067
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab
--principal=oryx/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:02.012496 9085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:02.012833 9085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:02.012884 9085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:02.017733 9085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:02.017841 9085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:02.017886 9085 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 12:24:02.017932 9085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:24:02.024034 9085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab
--principal=oryx/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.210.126:40067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:02.025779 9085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:02.027168 9085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:02.038815 9095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:02.038957 9097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:02.039006 9102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:02.039069 9085 server_base.cc:1034] running on GCE node
I20260806 12:24:02.039655 9085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:02.040495 9085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:02.041685 9085 hybrid_clock.cc:648] HybridClock initialized: now 1786019042041660 us; error 40 us; skew 500 ppm
Aug 06 12:24:02 dist-test-slave-lgr5 krb5kdc[8897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019042, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:02.045017 9085 init.cc:377] Logged in from keytab as oryx/127.28.210.65@KRBTEST.COM (short username oryx)
I20260806 12:24:02.046110 9085 webserver.cc:460] Webserver started at http://127.28.210.65:40607/ using document root <none> and password file <none>
I20260806 12:24:02.046737 9085 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:02.046824 9085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:02.047060 9085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:02.049484 9085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "df5a32b07c884f19bbf9b26799b48442"
format_stamp: "Formatted at 2026-08-06 12:24:02 on dist-test-slave-lgr5"
server_key: "1e948e4f3afca2b06f08ebe2384d40ad"
server_key_iv: "bdff07c7713b865c97e64fccc6cec9dd"
server_key_version: "encryptionkey@0"
I20260806 12:24:02.050144 9085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "df5a32b07c884f19bbf9b26799b48442"
format_stamp: "Formatted at 2026-08-06 12:24:02 on dist-test-slave-lgr5"
server_key: "1e948e4f3afca2b06f08ebe2384d40ad"
server_key_iv: "bdff07c7713b865c97e64fccc6cec9dd"
server_key_version: "encryptionkey@0"
I20260806 12:24:02.054608 9085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:24:02.057554 9109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:02.059098 9085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 12:24:02.059268 9085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "df5a32b07c884f19bbf9b26799b48442"
format_stamp: "Formatted at 2026-08-06 12:24:02 on dist-test-slave-lgr5"
server_key: "1e948e4f3afca2b06f08ebe2384d40ad"
server_key_iv: "bdff07c7713b865c97e64fccc6cec9dd"
server_key_version: "encryptionkey@0"
I20260806 12:24:02.059398 9085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:02.081930 9085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:02.086403 9085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:02.086666 9085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:02.088636 9085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:02.088711 9085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:02.088776 9085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:02.088807 9085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:02.100981 9085 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:34455
I20260806 12:24:02.101007 9291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:34455 every 8 connection(s)
Aug 06 12:24:02 dist-test-slave-lgr5 krb5kdc[8897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019042, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:24:02.102188 9085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:24:02.103777 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 9085
I20260806 12:24:02.104267 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:24:02.104669 29513 external_mini_cluster.cc:1428] Setting key 34bea46510d6889a4522c1c812676a87
I20260806 12:24:02.108577 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.089630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:47789 (local address 127.28.210.126:40067)
0806 12:24:02.089774 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.089778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.091146 (+ 1368us) server_negotiation.cc:407] Connection header received
0806 12:24:02.092409 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.092413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.092488 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.092574 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:02.094823 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.095658 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.097241 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.097506 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.097603 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.102529 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:02.102535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:02.102539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:02.102581 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:02.105151 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.106002 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.106006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.106007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.106166 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.106799 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.106803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.106805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.107125 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:02.107260 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.108181 (+ 921us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.108379 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 12:24:02.109534 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.090329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40067 (local address 127.28.210.65:47789)
0806 12:24:02.090949 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.091002 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.092043 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.092956 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.092974 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.093580 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:02.094509 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.094631 (+ 122us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.095804 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.095808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.097095 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.097108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.097674 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.097679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.097892 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.099150 (+ 1258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:02.099645 (+ 495us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:02.102162 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:02.105400 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:02.105408 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:02.105427 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:02.105829 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:02.106290 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:02.106293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:02.106295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:02.106592 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:02.107433 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:02.107440 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:02.107781 (+ 341us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.108018 (+ 237us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.108438 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":152,"threads_started":1}
I20260806 12:24:02.111179 9292 heartbeater.cc:347] Connected to a master server at 127.28.210.126:40067
I20260806 12:24:02.111393 9292 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:02.112231 9292 heartbeater.cc:511] Master 127.28.210.126:40067 requested a full tablet report, sending...
I20260806 12:24:02.114344 8971 ts_manager.cc:194] Registered new tserver with Master: df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455)
I20260806 12:24:02.115676 8971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:47789
WARNING: no policy specified for oryx/127.28.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.66@KRBTEST.COM" created.
I20260806 12:24:02.125378 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.116323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40067 (local address 127.28.210.65:32909)
0806 12:24:02.116858 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.116875 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.117124 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.117455 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.117458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.117696 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:02.117865 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.117876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.118903 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.118906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.119490 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.119499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.120014 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.120018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.120129 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.120667 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:02.120681 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:02.121358 (+ 677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:02.123122 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:02.123126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:02.123128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:02.123394 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:02.124023 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:02.124026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:02.124029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:02.124179 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:02.124710 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:02.124713 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:02.124917 (+ 204us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.125027 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.125194 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":168,"threads_started":1}
I20260806 12:24:02.125434 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.116441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:32909 (local address 127.28.210.126:40067)
0806 12:24:02.116597 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.116602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.117015 (+ 413us) server_negotiation.cc:407] Connection header received
0806 12:24:02.117265 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.117268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.117324 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.117415 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:02.118004 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.118761 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.119612 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.119888 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.119985 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.121550 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:02.121554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:02.121556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:02.121583 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:02.122995 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.123653 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.123659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.123661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.123859 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.124298 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.124302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.124304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.124575 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:02.124675 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.125134 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.125279 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab.
Entry for principal oryx/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab.
I20260806 12:24:02.139721 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:40067
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab
--principal=oryx/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:02.270685 9304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:02.271027 9304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:02.271075 9304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:02.275957 9304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:02.276105 9304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:02.276152 9304 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 12:24:02.276219 9304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:24:02.282856 9304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab
--principal=oryx/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.210.126:40067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:02.284327 9304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:02.285774 9304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:02.299021 9314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:02.299011 9313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:02.299021 9316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:02.299796 9304 server_base.cc:1034] running on GCE node
I20260806 12:24:02.300465 9304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:02.301419 9304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:02.302631 9304 hybrid_clock.cc:648] HybridClock initialized: now 1786019042302597 us; error 53 us; skew 500 ppm
Aug 06 12:24:02 dist-test-slave-lgr5 krb5kdc[8897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019042, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:02.306075 9304 init.cc:377] Logged in from keytab as oryx/127.28.210.66@KRBTEST.COM (short username oryx)
I20260806 12:24:02.307212 9304 webserver.cc:460] Webserver started at http://127.28.210.66:45337/ using document root <none> and password file <none>
I20260806 12:24:02.307880 9304 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:02.307961 9304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:02.308272 9304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:02.310811 9304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "16225fa0e41d46ad91f39b67765ce083"
format_stamp: "Formatted at 2026-08-06 12:24:02 on dist-test-slave-lgr5"
server_key: "61529cf8aed2cfd5e2d4e4dcdbc74997"
server_key_iv: "e6aeff838c764015b07a1a9de0564827"
server_key_version: "encryptionkey@0"
I20260806 12:24:02.311493 9304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "16225fa0e41d46ad91f39b67765ce083"
format_stamp: "Formatted at 2026-08-06 12:24:02 on dist-test-slave-lgr5"
server_key: "61529cf8aed2cfd5e2d4e4dcdbc74997"
server_key_iv: "e6aeff838c764015b07a1a9de0564827"
server_key_version: "encryptionkey@0"
I20260806 12:24:02.316653 9304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 12:24:02.319921 9326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:02.321785 9304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:24:02.321957 9304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "16225fa0e41d46ad91f39b67765ce083"
format_stamp: "Formatted at 2026-08-06 12:24:02 on dist-test-slave-lgr5"
server_key: "61529cf8aed2cfd5e2d4e4dcdbc74997"
server_key_iv: "e6aeff838c764015b07a1a9de0564827"
server_key_version: "encryptionkey@0"
I20260806 12:24:02.322106 9304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:02.376443 9304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:02.380363 9304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:02.380599 9304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:02.382350 9304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:02.382413 9304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:02.382488 9304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:02.382524 9304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 12:24:02 dist-test-slave-lgr5 krb5kdc[8897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019042, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:24:02.394006 9304 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:37315
I20260806 12:24:02.394192 9499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:37315 every 8 connection(s)
I20260806 12:24:02.395067 9304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:24:02.397332 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.383392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:42091 (local address 127.28.210.126:40067)
0806 12:24:02.383519 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.383523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.384754 (+ 1231us) server_negotiation.cc:407] Connection header received
0806 12:24:02.386016 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.386020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.386096 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.386192 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:02.387935 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.388820 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.390332 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.390688 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.390811 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.393546 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:02.393550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:02.393553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:02.393588 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:02.395042 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.395582 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.395586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.395588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.395686 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.396161 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.396164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.396166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.396391 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:02.396482 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.397074 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.397205 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 12:24:02.398219 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.383960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40067 (local address 127.28.210.66:42091)
0806 12:24:02.384571 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.384619 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.385682 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.386428 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.386439 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.386925 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:02.387709 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.387734 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.388977 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.388981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.390185 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.390199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.390886 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.390891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.391095 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.391771 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:02.391789 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:02.393342 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:02.395193 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:02.395199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:02.395214 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:02.395477 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:02.395790 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:02.395793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:02.395795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:02.395961 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:02.396522 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:02.396529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:02.396795 (+ 266us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.396988 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.397306 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":147,"threads_started":1}
I20260806 12:24:02.399555 9500 heartbeater.cc:347] Connected to a master server at 127.28.210.126:40067
I20260806 12:24:02.399721 9500 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:02.400156 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 9304
I20260806 12:24:02.400295 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:24:02.400323 9500 heartbeater.cc:511] Master 127.28.210.126:40067 requested a full tablet report, sending...
I20260806 12:24:02.400665 29513 external_mini_cluster.cc:1428] Setting key 4b78b6d284f8e5ffc8fecef6f1ed63bd
I20260806 12:24:02.401530 8972 ts_manager.cc:194] Registered new tserver with Master: 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66:37315)
I20260806 12:24:02.402305 8972 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:42091
I20260806 12:24:02.411338 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.401949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:50175 (local address 127.28.210.126:40067)
0806 12:24:02.402065 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.402070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.404406 (+ 2336us) server_negotiation.cc:407] Connection header received
0806 12:24:02.404524 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.404527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.404571 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.404634 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:02.405085 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.405584 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.406638 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.406863 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.406954 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.408289 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:02.408293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:02.408295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:02.408318 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:02.409626 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.410096 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.410099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.410100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.410176 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.410444 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.410446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.410448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.410704 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:02.410782 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.411105 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.411206 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 12:24:02.411432 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.401981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40067 (local address 127.28.210.66:50175)
0806 12:24:02.404315 (+ 2334us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.404330 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.404444 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.404662 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.404665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.404870 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:02.404975 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.404981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.405710 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.405714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.406493 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.406506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.406988 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.406992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.407084 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.407631 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:02.407648 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:02.408141 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:02.409739 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:02.409743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:02.409745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:02.409996 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:02.410262 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:02.410266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:02.410267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:02.410366 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:02.410796 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:02.410799 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:02.410942 (+ 143us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.411059 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.411217 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2227,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for oryx/127.28.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab.
Entry for principal oryx/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab.
I20260806 12:24:02.433035 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:40067
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab
--principal=oryx/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:02.553658 9514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:02.553977 9514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:02.554021 9514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:02.558187 9514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:02.558269 9514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:02.558310 9514 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 12:24:02.558342 9514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:24:02.564052 9514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab
--principal=oryx/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.210.126:40067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:02.565346 9514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:02.566430 9514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:02.577459 9519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:02.577615 9514 server_base.cc:1034] running on GCE node
W20260806 12:24:02.577419 9520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:02.577455 9522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:02.578694 9514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:02.579639 9514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:02.580998 9514 hybrid_clock.cc:648] HybridClock initialized: now 1786019042580929 us; error 91 us; skew 500 ppm
Aug 06 12:24:02 dist-test-slave-lgr5 krb5kdc[8897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019042, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:02.584510 9514 init.cc:377] Logged in from keytab as oryx/127.28.210.67@KRBTEST.COM (short username oryx)
I20260806 12:24:02.585619 9514 webserver.cc:460] Webserver started at http://127.28.210.67:38795/ using document root <none> and password file <none>
I20260806 12:24:02.586227 9514 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:02.586309 9514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:02.586536 9514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:02.588824 9514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "42a2f4f8f89849cd909d8faa51264fb2"
format_stamp: "Formatted at 2026-08-06 12:24:02 on dist-test-slave-lgr5"
server_key: "39687955144915a3e48fe9f571459250"
server_key_iv: "af31fe57d459cf18e4a6948bf5db0015"
server_key_version: "encryptionkey@0"
I20260806 12:24:02.589423 9514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "42a2f4f8f89849cd909d8faa51264fb2"
format_stamp: "Formatted at 2026-08-06 12:24:02 on dist-test-slave-lgr5"
server_key: "39687955144915a3e48fe9f571459250"
server_key_iv: "af31fe57d459cf18e4a6948bf5db0015"
server_key_version: "encryptionkey@0"
I20260806 12:24:02.593582 9514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:24:02.596426 9532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:02.598040 9514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:24:02.598165 9514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "42a2f4f8f89849cd909d8faa51264fb2"
format_stamp: "Formatted at 2026-08-06 12:24:02 on dist-test-slave-lgr5"
server_key: "39687955144915a3e48fe9f571459250"
server_key_iv: "af31fe57d459cf18e4a6948bf5db0015"
server_key_version: "encryptionkey@0"
I20260806 12:24:02.598296 9514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:02.612711 9514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:02.615826 9514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:02.616003 9514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:02.617623 9514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:02.617682 9514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:02.617728 9514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:02.617758 9514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 12:24:02 dist-test-slave-lgr5 krb5kdc[8897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019042, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:24:02.628973 9514 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:34261
I20260806 12:24:02.629038 9708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:34261 every 8 connection(s)
I20260806 12:24:02.630300 9514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:24:02.631229 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 9514
I20260806 12:24:02.631393 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:24:02.631763 29513 external_mini_cluster.cc:1428] Setting key 1342537f3e633f89cea5c3df5b6fb87a
I20260806 12:24:02.634279 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.618543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:42541 (local address 127.28.210.126:40067)
0806 12:24:02.618665 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.618669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.619857 (+ 1188us) server_negotiation.cc:407] Connection header received
0806 12:24:02.621120 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.621124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.621191 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.621268 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:02.623146 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.623982 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.625844 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.626081 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.626192 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.629568 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:02.629573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:02.629575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:02.629615 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:02.631368 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.632035 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.632039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.632041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.632191 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.632703 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.632707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.632709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.633017 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:02.633120 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.633913 (+ 793us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.634091 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 12:24:02.635351 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.619135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40067 (local address 127.28.210.67:42541)
0806 12:24:02.619683 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.619729 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.620846 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.621558 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.621568 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.622102 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:02.622934 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.622962 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.624160 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.624165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.625688 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.625701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.626264 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.626269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.626498 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.627293 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:02.627317 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:02.629289 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:02.631503 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:02.631511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:02.631530 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:02.631916 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:02.632322 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:02.632325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:02.632328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:02.632566 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:02.633149 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:02.633157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:02.633574 (+ 417us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.633786 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.634151 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":139,"threads_started":1}
I20260806 12:24:02.637122 9710 heartbeater.cc:347] Connected to a master server at 127.28.210.126:40067
I20260806 12:24:02.637334 9710 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:02.638170 9710 heartbeater.cc:511] Master 127.28.210.126:40067 requested a full tablet report, sending...
I20260806 12:24:02.639632 8972 ts_manager.cc:194] Registered new tserver with Master: 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67:34261)
I20260806 12:24:02.640427 8972 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:42541
I20260806 12:24:02.647456 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:24:02.649775 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.640633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:54695 (local address 127.28.210.126:40067)
0806 12:24:02.640786 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.640791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.641350 (+ 559us) server_negotiation.cc:407] Connection header received
0806 12:24:02.641661 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.641666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.641752 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.641866 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:02.642503 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.643056 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.643933 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.644229 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.644319 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.646087 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:02.646107 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:02.646112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:02.646157 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:02.647868 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.648463 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.648466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.648468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.648566 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.648848 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.648851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.648853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.649067 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:02.649167 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.649519 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.649641 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 12:24:02.649842 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.640512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40067 (local address 127.28.210.67:54695)
0806 12:24:02.641035 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.641049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.641425 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.641920 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.641923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.642176 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:02.642305 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.642313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.643212 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.643216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.643805 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.643815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.644342 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.644345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.644444 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.645160 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:02.645177 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:02.645807 (+ 630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:02.648028 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:02.648031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:02.648033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:02.648358 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:02.648661 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:02.648664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:02.648666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:02.648755 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:02.649170 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:02.649173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:02.649305 (+ 132us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.649447 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.649632 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":162,"threads_started":1}
Aug 06 12:24:02 dist-test-slave-lgr5 krb5kdc[8897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 12:24:02 dist-test-slave-lgr5 krb5kdc[8897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:02 dist-test-slave-lgr5 krb5kdc[8897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 12:24:02.677143 9043 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:02.670613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.28.210.126:40067)
0806 12:24:02.670755 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.670759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.670772 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:24:02.670817 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.670820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.670863 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.670926 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:02.671731 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.672358 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.673231 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.673476 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.673556 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.676994 (+ 3438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52472: BlockingRecv error: recv got EOF from 127.0.0.1:52472 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 12:24:02 dist-test-slave-lgr5 krb5kdc[8897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:24:02.686988 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.678691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.28.210.126:40067)
0806 12:24:02.678813 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.678817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.678941 (+ 124us) server_negotiation.cc:407] Connection header received
0806 12:24:02.679074 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.679076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.679118 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.679178 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:02.679745 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.680339 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.681187 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.681404 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.681491 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.683907 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:02.683911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:02.683913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:02.683942 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:02.685315 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.685823 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.685828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.685830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.685915 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:02.686214 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:02.686217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:02.686218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:02.686429 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:02.686538 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.686743 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.686856 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 12:24:02.690352 8972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:24:02.693711 8972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:24:02.710901 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.704900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.28.210.66:37315)
0806 12:24:02.705452 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.705464 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.705711 (+ 247us) server_negotiation.cc:407] Connection header received
0806 12:24:02.705827 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.705832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.705998 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.706216 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:24:02.706579 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.707363 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.708988 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.709897 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.710028 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.710229 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.710441 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.710612 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":137,"threads_started":1}
I20260806 12:24:02.710824 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.704157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:37315 (local address 127.0.0.1:56152)
0806 12:24:02.705594 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.705607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.705712 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.706363 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.706366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.706382 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:24:02.706464 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.706469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.707511 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.707515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.708834 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.708848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.710057 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.710061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.710188 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.710237 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.710397 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.710615 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"thread_start_us":131,"threads_started":1}
I20260806 12:24:02.711292 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.704764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.28.210.67:34261)
0806 12:24:02.705452 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.705464 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.705490 (+ 26us) server_negotiation.cc:407] Connection header received
0806 12:24:02.705583 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.705588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.705837 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.706077 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:24:02.706406 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.707918 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.709366 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.709950 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.710058 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.710258 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.710966 (+ 708us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.711113 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":141,"threads_started":1}
I20260806 12:24:02.710824 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.703874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:34261 (local address 127.0.0.1:59304)
0806 12:24:02.705225 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.705292 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.705449 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.706093 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.706100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.706126 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:24:02.706282 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.706288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.708051 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.708055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.709238 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.709247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.710323 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.710327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.710444 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.710465 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.710534 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.710600 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"mutex_wait_us":4,"thread_start_us":126,"threads_started":1}
I20260806 12:24:02.711922 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.704368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.28.210.65:34455)
0806 12:24:02.704753 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.704765 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.705404 (+ 639us) server_negotiation.cc:407] Connection header received
0806 12:24:02.705622 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.705627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.705840 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.705962 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:24:02.706465 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.707623 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.709259 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.710069 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.710166 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.710316 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.710935 (+ 619us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.711790 (+ 855us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":109,"threads_started":1}
I20260806 12:24:02.712430 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.703942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:34455 (local address 127.0.0.1:49922)
0806 12:24:02.705307 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.705320 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.705467 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.706180 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.706183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.706204 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:24:02.706330 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.706338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.707764 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.707768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.709096 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.709108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.710385 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.710389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.710499 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.710524 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.710640 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.710829 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1300,"mutex_wait_us":26,"thread_start_us":110,"threads_started":1}
I20260806 12:24:02.715410 9402 tablet_service.cc:1467] Processing CreateTablet for tablet a5b60dfc286c4b04bc8cb9a95720ad34 (DEFAULT_TABLE table=test-table [id=f9f6ff21ac4d4b13addbd055c1cdac4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:02.715378 9183 tablet_service.cc:1467] Processing CreateTablet for tablet a5b60dfc286c4b04bc8cb9a95720ad34 (DEFAULT_TABLE table=test-table [id=f9f6ff21ac4d4b13addbd055c1cdac4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:02.716974 9402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b60dfc286c4b04bc8cb9a95720ad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:02.716974 9183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b60dfc286c4b04bc8cb9a95720ad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:02.718202 9606 tablet_service.cc:1467] Processing CreateTablet for tablet a5b60dfc286c4b04bc8cb9a95720ad34 (DEFAULT_TABLE table=test-table [id=f9f6ff21ac4d4b13addbd055c1cdac4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:02.719357 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b60dfc286c4b04bc8cb9a95720ad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:02.728178 9753 tablet_bootstrap.cc:492] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083: Bootstrap starting.
I20260806 12:24:02.728837 9755 tablet_bootstrap.cc:492] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2: Bootstrap starting.
I20260806 12:24:02.731139 9755 tablet_bootstrap.cc:654] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:02.731139 9753 tablet_bootstrap.cc:654] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:02.728067 9750 tablet_bootstrap.cc:492] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442: Bootstrap starting.
I20260806 12:24:02.732235 9753 log.cc:824] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083: Log is configured to *not* fsync() on all Append() calls
I20260806 12:24:02.732234 9755 log.cc:824] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2: Log is configured to *not* fsync() on all Append() calls
I20260806 12:24:02.734614 9750 tablet_bootstrap.cc:654] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:02.734825 9755 tablet_bootstrap.cc:492] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2: No bootstrap required, opened a new log
I20260806 12:24:02.734825 9753 tablet_bootstrap.cc:492] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083: No bootstrap required, opened a new log
I20260806 12:24:02.735080 9753 ts_tablet_manager.cc:1397] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 12:24:02.735080 9755 ts_tablet_manager.cc:1397] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 12:24:02.735752 9750 log.cc:824] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442: Log is configured to *not* fsync() on all Append() calls
I20260806 12:24:02.738032 9750 tablet_bootstrap.cc:492] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442: No bootstrap required, opened a new log
I20260806 12:24:02.738263 9750 ts_tablet_manager.cc:1397] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260806 12:24:02.739359 9753 raft_consensus.cc:348] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } }
I20260806 12:24:02.739359 9755 raft_consensus.cc:348] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } }
I20260806 12:24:02.739692 9755 raft_consensus.cc:374] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:02.739691 9753 raft_consensus.cc:374] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:02.739753 9753 raft_consensus.cc:729] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16225fa0e41d46ad91f39b67765ce083, State: Initialized, Role: FOLLOWER
I20260806 12:24:02.739753 9755 raft_consensus.cc:729] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a2f4f8f89849cd909d8faa51264fb2, State: Initialized, Role: FOLLOWER
I20260806 12:24:02.740298 9755 consensus_queue.cc:260] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } }
I20260806 12:24:02.740298 9753 consensus_queue.cc:260] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } }
I20260806 12:24:02.741345 9710 heartbeater.cc:503] Master 127.28.210.126:40067 was elected leader, sending a full tablet report...
I20260806 12:24:02.741511 9753 ts_tablet_manager.cc:1428] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 12:24:02.741511 9755 ts_tablet_manager.cc:1428] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 12:24:02.741895 9500 heartbeater.cc:503] Master 127.28.210.126:40067 was elected leader, sending a full tablet report...
I20260806 12:24:02.742555 9750 raft_consensus.cc:348] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } }
I20260806 12:24:02.742841 9750 raft_consensus.cc:374] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:02.742913 9750 raft_consensus.cc:729] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5a32b07c884f19bbf9b26799b48442, State: Initialized, Role: FOLLOWER
I20260806 12:24:02.743423 9750 consensus_queue.cc:260] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } }
I20260806 12:24:02.744241 9292 heartbeater.cc:503] Master 127.28.210.126:40067 was elected leader, sending a full tablet report...
I20260806 12:24:02.744593 9750 ts_tablet_manager.cc:1428] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 12:24:02.763645 9760 raft_consensus.cc:482] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:24:02.763932 9760 raft_consensus.cc:504] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } }
I20260806 12:24:02.765883 9760 leader_election.cc:288] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67:34261), df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455)
I20260806 12:24:02.769055 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.765587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:34455 (local address 127.28.210.66:57927)
0806 12:24:02.765711 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.765728 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.765851 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.766185 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.766188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.766212 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:24:02.766319 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.766324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.767082 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.767085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.767892 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.767901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.768611 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.768615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.768731 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.768755 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.768832 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.768907 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 12:24:02.769100 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.765678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:57927 (local address 127.28.210.65:34455)
0806 12:24:02.765819 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.765823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.765843 (+ 20us) server_negotiation.cc:407] Connection header received
0806 12:24:02.766004 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.766007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.766062 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.766136 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:24:02.766439 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.766968 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.768022 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.768495 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.768610 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.768670 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.768872 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.768966 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 12:24:02.769865 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.765595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:34261 (local address 127.28.210.66:56563)
0806 12:24:02.766150 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.766162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.766381 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.766605 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.766608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.766623 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:24:02.767093 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.767100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.767830 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.767834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.768615 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.768625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.769460 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.769464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.769553 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.769577 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.769651 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.769716 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":141,"thread_start_us":134,"threads_started":1}
I20260806 12:24:02.769856 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.765809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:56563 (local address 127.28.210.67:34261)
0806 12:24:02.765944 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.765948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.766351 (+ 403us) server_negotiation.cc:407] Connection header received
0806 12:24:02.766481 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.766483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.766523 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.766585 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:24:02.767215 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.767716 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.768752 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.769311 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.769432 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.769479 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.769675 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.769748 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 12:24:02.770257 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b60dfc286c4b04bc8cb9a95720ad34" candidate_uuid: "16225fa0e41d46ad91f39b67765ce083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a32b07c884f19bbf9b26799b48442" is_pre_election: true
I20260806 12:24:02.770632 9214 raft_consensus.cc:2454] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16225fa0e41d46ad91f39b67765ce083 in term 0.
I20260806 12:24:02.770629 9637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b60dfc286c4b04bc8cb9a95720ad34" candidate_uuid: "16225fa0e41d46ad91f39b67765ce083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a2f4f8f89849cd909d8faa51264fb2" is_pre_election: true
I20260806 12:24:02.770890 9637 raft_consensus.cc:2454] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16225fa0e41d46ad91f39b67765ce083 in term 0.
I20260806 12:24:02.771240 9329 leader_election.cc:302] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16225fa0e41d46ad91f39b67765ce083, df5a32b07c884f19bbf9b26799b48442; no voters:
I20260806 12:24:02.771576 9760 raft_consensus.cc:2781] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:24:02.771677 9760 raft_consensus.cc:482] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:24:02.771715 9760 raft_consensus.cc:3037] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:02.773149 9760 raft_consensus.cc:504] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } }
I20260806 12:24:02.773562 9760 leader_election.cc:288] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [CANDIDATE]: Term 1 election: Requested vote from peers 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67:34261), df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455)
I20260806 12:24:02.773938 9637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b60dfc286c4b04bc8cb9a95720ad34" candidate_uuid: "16225fa0e41d46ad91f39b67765ce083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a2f4f8f89849cd909d8faa51264fb2"
I20260806 12:24:02.774011 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b60dfc286c4b04bc8cb9a95720ad34" candidate_uuid: "16225fa0e41d46ad91f39b67765ce083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a32b07c884f19bbf9b26799b48442"
I20260806 12:24:02.774091 9637 raft_consensus.cc:3037] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:02.774142 9214 raft_consensus.cc:3037] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:02.775480 9214 raft_consensus.cc:2454] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16225fa0e41d46ad91f39b67765ce083 in term 1.
I20260806 12:24:02.775482 9637 raft_consensus.cc:2454] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16225fa0e41d46ad91f39b67765ce083 in term 1.
I20260806 12:24:02.775908 9329 leader_election.cc:302] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16225fa0e41d46ad91f39b67765ce083, df5a32b07c884f19bbf9b26799b48442; no voters:
I20260806 12:24:02.776145 9760 raft_consensus.cc:2781] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:24:02.776398 9760 raft_consensus.cc:686] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 1 LEADER]: Becoming Leader. State: Replica: 16225fa0e41d46ad91f39b67765ce083, State: Running, Role: LEADER
I20260806 12:24:02.776772 9760 consensus_queue.cc:237] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } }
I20260806 12:24:02.780679 8969 catalog_manager.cc:5495] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66). New cstate: current_term: 1 leader_uuid: "16225fa0e41d46ad91f39b67765ce083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } health_report { overall_health: HEALTHY } } }
I20260806 12:24:02.794638 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.790596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.28.210.66:37315)
0806 12:24:02.790719 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.790722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.790737 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:24:02.790849 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.790851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.790911 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.791013 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:24:02.791310 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.791853 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.792927 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.793200 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.793298 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.793677 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:24:02.794234 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:24:02.794326 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.794412 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.794491 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 12:24:02.802218 9637 raft_consensus.cc:1261] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Refusing update from remote peer 16225fa0e41d46ad91f39b67765ce083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:02.803135 9214 raft_consensus.cc:1261] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Refusing update from remote peer 16225fa0e41d46ad91f39b67765ce083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:02.803176 9766 consensus_queue.cc:1035] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:02.803828 9760 consensus_queue.cc:1035] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:02.813139 9772 mvcc.cc:205] Tried to move back new op lower bound from 7315533999307993088 to 7315533999216799744. Current Snapshot: MvccSnapshot[applied={T|T < 7315533999307993088}]
I20260806 12:24:02.813694 9771 mvcc.cc:205] Tried to move back new op lower bound from 7315533999307993088 to 7315533999216799744. Current Snapshot: MvccSnapshot[applied={T|T < 7315533999307993088}]
I20260806 12:24:02.824285 8972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:24:02.824522 8972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:24:02.827461 8972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa: Sending DeleteTablet for 3 replicas of tablet a5b60dfc286c4b04bc8cb9a95720ad34
I20260806 12:24:02.828668 9402 tablet_service.cc:1514] Processing DeleteTablet for tablet a5b60dfc286c4b04bc8cb9a95720ad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:02 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:56152
I20260806 12:24:02.828668 9183 tablet_service.cc:1514] Processing DeleteTablet for tablet a5b60dfc286c4b04bc8cb9a95720ad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:02 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:49922
I20260806 12:24:02.828745 9606 tablet_service.cc:1514] Processing DeleteTablet for tablet a5b60dfc286c4b04bc8cb9a95720ad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:02 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:59304
I20260806 12:24:02.829540 9782 tablet_replica.cc:330] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442: stopping tablet replica
I20260806 12:24:02.829536 9783 tablet_replica.cc:330] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083: stopping tablet replica
I20260806 12:24:02.829959 9782 raft_consensus.cc:2229] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:24:02.829959 9783 raft_consensus.cc:2229] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:24:02.830446 9782 raft_consensus.cc:2258] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:02.830440 9783 raft_consensus.cc:2258] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:02.831038 9785 tablet_replica.cc:330] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2: stopping tablet replica
I20260806 12:24:02.831395 9785 raft_consensus.cc:2229] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:24:02.831308 8972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 12:24:02.831884 9785 raft_consensus.cc:2258] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 12:24:02.831892 8972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:24:02.833062 9782 ts_tablet_manager.cc:1905] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:02.833793 9785 ts_tablet_manager.cc:1905] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:02.837507 9782 ts_tablet_manager.cc:1918] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:24:02.837579 9783 ts_tablet_manager.cc:1905] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:02.837883 9782 log.cc:1196] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/wal/wals/a5b60dfc286c4b04bc8cb9a95720ad34
I20260806 12:24:02.838511 9782 ts_tablet_manager.cc:1939] T a5b60dfc286c4b04bc8cb9a95720ad34 P df5a32b07c884f19bbf9b26799b48442: Deleting consensus metadata
I20260806 12:24:02.839600 9183 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9a154dfab64d8db791f8fba40bc441 (DEFAULT_TABLE table=test-table [id=e619e7da00bd48e4beabb2e25b387dd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:02.839603 9402 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9a154dfab64d8db791f8fba40bc441 (DEFAULT_TABLE table=test-table [id=e619e7da00bd48e4beabb2e25b387dd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:02.839934 9183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9a154dfab64d8db791f8fba40bc441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:02.839959 9402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9a154dfab64d8db791f8fba40bc441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:02.840811 8950 catalog_manager.cc:4841] TS df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455): tablet a5b60dfc286c4b04bc8cb9a95720ad34 (table test-table [id=f9f6ff21ac4d4b13addbd055c1cdac4e]) successfully deleted
I20260806 12:24:02.842608 9785 ts_tablet_manager.cc:1918] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:24:02.842952 9750 tablet_bootstrap.cc:492] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442: Bootstrap starting.
I20260806 12:24:02.844291 9750 tablet_bootstrap.cc:654] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:02.843986 9606 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9a154dfab64d8db791f8fba40bc441 (DEFAULT_TABLE table=test-table [id=e619e7da00bd48e4beabb2e25b387dd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:02.844861 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9a154dfab64d8db791f8fba40bc441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:02.845530 9753 tablet_bootstrap.cc:492] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083: Bootstrap starting.
I20260806 12:24:02.846654 9753 tablet_bootstrap.cc:654] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:02.846767 9750 tablet_bootstrap.cc:492] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442: No bootstrap required, opened a new log
I20260806 12:24:02.846874 9750 ts_tablet_manager.cc:1397] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 12:24:02.847886 9755 tablet_bootstrap.cc:492] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2: Bootstrap starting.
I20260806 12:24:02.847857 9750 raft_consensus.cc:348] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:02.848011 9750 raft_consensus.cc:374] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:02.848053 9750 raft_consensus.cc:729] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5a32b07c884f19bbf9b26799b48442, State: Initialized, Role: FOLLOWER
I20260806 12:24:02.848280 9750 consensus_queue.cc:260] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:02.848544 9785 log.cc:1196] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/a5b60dfc286c4b04bc8cb9a95720ad34
I20260806 12:24:02.848578 9750 ts_tablet_manager.cc:1428] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 12:24:02.848996 9785 ts_tablet_manager.cc:1939] T a5b60dfc286c4b04bc8cb9a95720ad34 P 42a2f4f8f89849cd909d8faa51264fb2: Deleting consensus metadata
I20260806 12:24:02.849045 9755 tablet_bootstrap.cc:654] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:02.851392 8951 catalog_manager.cc:4841] TS 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67:34261): tablet a5b60dfc286c4b04bc8cb9a95720ad34 (table test-table [id=f9f6ff21ac4d4b13addbd055c1cdac4e]) successfully deleted
I20260806 12:24:02.851550 9755 tablet_bootstrap.cc:492] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2: No bootstrap required, opened a new log
I20260806 12:24:02.851650 9755 ts_tablet_manager.cc:1397] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 12:24:02.852293 9783 ts_tablet_manager.cc:1918] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:24:02.852419 9783 log.cc:1196] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/a5b60dfc286c4b04bc8cb9a95720ad34
I20260806 12:24:02.852891 9753 tablet_bootstrap.cc:492] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083: No bootstrap required, opened a new log
I20260806 12:24:02.852949 9783 ts_tablet_manager.cc:1939] T a5b60dfc286c4b04bc8cb9a95720ad34 P 16225fa0e41d46ad91f39b67765ce083: Deleting consensus metadata
I20260806 12:24:02.853008 9753 ts_tablet_manager.cc:1397] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260806 12:24:02.853423 9755 raft_consensus.cc:348] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:02.853570 9755 raft_consensus.cc:374] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:02.853601 9755 raft_consensus.cc:729] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a2f4f8f89849cd909d8faa51264fb2, State: Initialized, Role: FOLLOWER
I20260806 12:24:02.853632 9753 raft_consensus.cc:348] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:02.853785 9753 raft_consensus.cc:374] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:02.853786 9755 consensus_queue.cc:260] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:02.853833 9753 raft_consensus.cc:729] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16225fa0e41d46ad91f39b67765ce083, State: Initialized, Role: FOLLOWER
I20260806 12:24:02.853994 9753 consensus_queue.cc:260] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:02.854347 9753 ts_tablet_manager.cc:1428] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260806 12:24:02.854645 9293 tablet.cc:2369] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:24:02.855633 8947 catalog_manager.cc:4841] TS 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66:37315): tablet a5b60dfc286c4b04bc8cb9a95720ad34 (table test-table [id=f9f6ff21ac4d4b13addbd055c1cdac4e]) successfully deleted
I20260806 12:24:02.856069 9755 ts_tablet_manager.cc:1428] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20260806 12:24:02.882222 9711 tablet.cc:2369] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:24:02.896538 9502 tablet.cc:2369] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:24:02.983592 9762 raft_consensus.cc:482] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:24:02.983798 9762 raft_consensus.cc:504] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:02.985044 9762 leader_election.cc:288] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455), 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66:37315)
I20260806 12:24:02.989400 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.985671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:58901 (local address 127.28.210.65:34455)
0806 12:24:02.985800 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.985805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.986117 (+ 312us) server_negotiation.cc:407] Connection header received
0806 12:24:02.986333 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.986336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.986389 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.986463 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:24:02.986735 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.987311 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.988413 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.988846 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.988960 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.989023 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.989199 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.989277 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 12:24:02.989418 9791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.985340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:34455 (local address 127.28.210.67:58901)
0806 12:24:02.986016 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.986030 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.986194 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.986485 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.986488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.986505 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:24:02.986628 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.986633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.987416 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.987419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.988285 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.988296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.988982 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.988985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.989080 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.989100 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.989166 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.989238 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":346,"thread_start_us":78,"threads_started":1}
I20260806 12:24:02.989555 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.985587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:46441 (local address 127.28.210.66:37315)
0806 12:24:02.985716 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:02.985721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:02.985736 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:24:02.985792 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:02.985794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:02.985860 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:02.985934 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:24:02.986280 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.986975 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.988485 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.988999 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.989114 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.989169 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:02.989354 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 12:24:02.989426 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 12:24:02.989555 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:02.985339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:37315 (local address 127.28.210.67:46441)
0806 12:24:02.985486 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:02.985502 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:02.985608 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:02.986038 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:02.986041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:02.986057 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:24:02.986169 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.986175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.987085 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.987088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.988330 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:02.988342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:02.989128 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:02.989131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:02.989240 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:02.989257 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 12:24:02.989325 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 12:24:02.989402 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":23}
I20260806 12:24:02.990345 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9a154dfab64d8db791f8fba40bc441" candidate_uuid: "42a2f4f8f89849cd909d8faa51264fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a32b07c884f19bbf9b26799b48442" is_pre_election: true
I20260806 12:24:02.990435 9425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9a154dfab64d8db791f8fba40bc441" candidate_uuid: "42a2f4f8f89849cd909d8faa51264fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16225fa0e41d46ad91f39b67765ce083" is_pre_election: true
I20260806 12:24:02.990523 9214 raft_consensus.cc:2454] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42a2f4f8f89849cd909d8faa51264fb2 in term 0.
I20260806 12:24:02.990680 9425 raft_consensus.cc:2454] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42a2f4f8f89849cd909d8faa51264fb2 in term 0.
I20260806 12:24:02.990902 9535 leader_election.cc:302] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a2f4f8f89849cd909d8faa51264fb2, df5a32b07c884f19bbf9b26799b48442; no voters:
I20260806 12:24:02.991199 9762 raft_consensus.cc:2781] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:24:02.991267 9762 raft_consensus.cc:482] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:24:02.991312 9762 raft_consensus.cc:3037] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:02.992440 9762 raft_consensus.cc:504] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:02.992796 9762 leader_election.cc:288] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [CANDIDATE]: Term 1 election: Requested vote from peers df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455), 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66:37315)
I20260806 12:24:02.993216 9425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9a154dfab64d8db791f8fba40bc441" candidate_uuid: "42a2f4f8f89849cd909d8faa51264fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16225fa0e41d46ad91f39b67765ce083"
I20260806 12:24:02.993216 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9a154dfab64d8db791f8fba40bc441" candidate_uuid: "42a2f4f8f89849cd909d8faa51264fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a32b07c884f19bbf9b26799b48442"
I20260806 12:24:02.993366 9425 raft_consensus.cc:3037] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:02.993366 9214 raft_consensus.cc:3037] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:02.994700 9214 raft_consensus.cc:2454] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42a2f4f8f89849cd909d8faa51264fb2 in term 1.
I20260806 12:24:02.994700 9425 raft_consensus.cc:2454] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42a2f4f8f89849cd909d8faa51264fb2 in term 1.
I20260806 12:24:02.995030 9535 leader_election.cc:302] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a2f4f8f89849cd909d8faa51264fb2, df5a32b07c884f19bbf9b26799b48442; no voters:
I20260806 12:24:02.995277 9762 raft_consensus.cc:2781] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:24:02.995517 9762 raft_consensus.cc:686] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 42a2f4f8f89849cd909d8faa51264fb2, State: Running, Role: LEADER
I20260806 12:24:02.995801 9762 consensus_queue.cc:237] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:02.998598 8972 catalog_manager.cc:5495] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67). New cstate: current_term: 1 leader_uuid: "42a2f4f8f89849cd909d8faa51264fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } health_report { overall_health: HEALTHY } } }
Aug 06 12:24:03 dist-test-slave-lgr5 krb5kdc[8897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019041, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.126@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:24:03.047765 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:03.038150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40067 (local address 127.0.0.1:52490)
0806 12:24:03.038733 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:03.038746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:03.038865 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:03.039084 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:03.039086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:03.039503 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:03.039611 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:03.039617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.040547 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.040550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:03.041407 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:03.041419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.041916 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.041919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:03.041988 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:03.042634 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:03.042648 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:03.044226 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:03.046054 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:03.046066 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:03.046073 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:03.046345 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:03.046637 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:03.046639 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:03.046641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:03.046727 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:03.047202 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:03.047209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:03.047351 (+ 142us) client_negotiation.cc:769] Sending connection context
0806 12:24:03.047458 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 12:24:03.047602 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":130,"threads_started":1}
I20260806 12:24:03.047866 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:03.038204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.28.210.126:40067)
0806 12:24:03.038371 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:03.038376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:03.038853 (+ 477us) server_negotiation.cc:407] Connection header received
0806 12:24:03.038935 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:03.038938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:03.038982 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:03.039055 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:03.039738 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.040376 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.041533 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.041781 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.041871 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:03.044425 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:03.044429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:03.044431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:03.044456 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:03.045922 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:03.046449 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:03.046452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:03.046453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:03.046552 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:03.046834 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:03.046838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:03.046839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:03.047067 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:03.047154 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:03.047570 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 12:24:03.047695 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 12:24:03.052764 8972 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:52490:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 12:24:03.053972 8972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:24:03.060523 9183 tablet_service.cc:1467] Processing CreateTablet for tablet a7347a5bdb3e42cdbd0e6a16feae297d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed84b46c712544949629cfdfc7cf17d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 12:24:03.060523 9402 tablet_service.cc:1467] Processing CreateTablet for tablet a7347a5bdb3e42cdbd0e6a16feae297d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed84b46c712544949629cfdfc7cf17d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 12:24:03.061057 9402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7347a5bdb3e42cdbd0e6a16feae297d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:03.061070 9183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7347a5bdb3e42cdbd0e6a16feae297d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:03.060915 9606 tablet_service.cc:1467] Processing CreateTablet for tablet a7347a5bdb3e42cdbd0e6a16feae297d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed84b46c712544949629cfdfc7cf17d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 12:24:03.061389 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7347a5bdb3e42cdbd0e6a16feae297d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:03.064447 9755 tablet_bootstrap.cc:492] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2: Bootstrap starting.
I20260806 12:24:03.064488 9753 tablet_bootstrap.cc:492] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: Bootstrap starting.
I20260806 12:24:03.064544 9750 tablet_bootstrap.cc:492] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442: Bootstrap starting.
I20260806 12:24:03.065459 9755 tablet_bootstrap.cc:654] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:03.065479 9753 tablet_bootstrap.cc:654] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:03.065479 9750 tablet_bootstrap.cc:654] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:03.067273 9755 tablet_bootstrap.cc:492] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2: No bootstrap required, opened a new log
I20260806 12:24:03.067267 9753 tablet_bootstrap.cc:492] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: No bootstrap required, opened a new log
I20260806 12:24:03.067379 9753 ts_tablet_manager.cc:1397] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:24:03.067379 9755 ts_tablet_manager.cc:1397] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:24:03.067476 9750 tablet_bootstrap.cc:492] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442: No bootstrap required, opened a new log
I20260806 12:24:03.067580 9750 ts_tablet_manager.cc:1397] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:24:03.067843 9753 raft_consensus.cc:348] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.067852 9755 raft_consensus.cc:348] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.067984 9755 raft_consensus.cc:374] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:03.067984 9753 raft_consensus.cc:374] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:03.068018 9755 raft_consensus.cc:729] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a2f4f8f89849cd909d8faa51264fb2, State: Initialized, Role: FOLLOWER
I20260806 12:24:03.068018 9753 raft_consensus.cc:729] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16225fa0e41d46ad91f39b67765ce083, State: Initialized, Role: FOLLOWER
I20260806 12:24:03.067996 9750 raft_consensus.cc:348] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.068149 9750 raft_consensus.cc:374] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:03.068189 9750 raft_consensus.cc:729] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5a32b07c884f19bbf9b26799b48442, State: Initialized, Role: FOLLOWER
I20260806 12:24:03.068194 9753 consensus_queue.cc:260] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.068214 9755 consensus_queue.cc:260] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.068325 9750 consensus_queue.cc:260] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.068503 9753 ts_tablet_manager.cc:1428] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 12:24:03.068473 9792 tablet_replica.cc:439] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } }
I20260806 12:24:03.068634 9792 tablet_replica.cc:442] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:24:03.068655 9750 ts_tablet_manager.cc:1428] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 12:24:03.068830 9775 tablet_replica.cc:439] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } }
I20260806 12:24:03.068979 9775 tablet_replica.cc:442] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:24:03.069268 9763 tablet_replica.cc:439] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } }
I20260806 12:24:03.069435 9763 tablet_replica.cc:442] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:24:03.070045 9755 ts_tablet_manager.cc:1428] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260806 12:24:03.107059 9775 raft_consensus.cc:482] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:24:03.107226 9775 raft_consensus.cc:504] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.107688 9775 leader_election.cc:288] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455), 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67:34261)
I20260806 12:24:03.108196 9637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7347a5bdb3e42cdbd0e6a16feae297d" candidate_uuid: "16225fa0e41d46ad91f39b67765ce083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a2f4f8f89849cd909d8faa51264fb2" is_pre_election: true
I20260806 12:24:03.108196 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7347a5bdb3e42cdbd0e6a16feae297d" candidate_uuid: "16225fa0e41d46ad91f39b67765ce083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a32b07c884f19bbf9b26799b48442" is_pre_election: true
I20260806 12:24:03.108354 9637 raft_consensus.cc:2454] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16225fa0e41d46ad91f39b67765ce083 in term 0.
I20260806 12:24:03.108354 9214 raft_consensus.cc:2454] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16225fa0e41d46ad91f39b67765ce083 in term 0.
I20260806 12:24:03.108667 9331 leader_election.cc:302] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16225fa0e41d46ad91f39b67765ce083, 42a2f4f8f89849cd909d8faa51264fb2; no voters:
I20260806 12:24:03.108861 9775 raft_consensus.cc:2781] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:24:03.108922 9775 raft_consensus.cc:482] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:24:03.108963 9775 raft_consensus.cc:3037] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:03.109972 9775 raft_consensus.cc:504] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.110309 9775 leader_election.cc:288] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [CANDIDATE]: Term 1 election: Requested vote from peers df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455), 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67:34261)
I20260806 12:24:03.110719 9637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7347a5bdb3e42cdbd0e6a16feae297d" candidate_uuid: "16225fa0e41d46ad91f39b67765ce083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a2f4f8f89849cd909d8faa51264fb2"
I20260806 12:24:03.110710 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7347a5bdb3e42cdbd0e6a16feae297d" candidate_uuid: "16225fa0e41d46ad91f39b67765ce083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a32b07c884f19bbf9b26799b48442"
I20260806 12:24:03.110838 9637 raft_consensus.cc:3037] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:03.110838 9214 raft_consensus.cc:3037] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:03.111872 9637 raft_consensus.cc:2454] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16225fa0e41d46ad91f39b67765ce083 in term 1.
I20260806 12:24:03.111977 9214 raft_consensus.cc:2454] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16225fa0e41d46ad91f39b67765ce083 in term 1.
I20260806 12:24:03.112203 9331 leader_election.cc:302] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16225fa0e41d46ad91f39b67765ce083, 42a2f4f8f89849cd909d8faa51264fb2; no voters:
I20260806 12:24:03.112428 9775 raft_consensus.cc:2781] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:24:03.112517 9775 raft_consensus.cc:686] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [term 1 LEADER]: Becoming Leader. State: Replica: 16225fa0e41d46ad91f39b67765ce083, State: Running, Role: LEADER
I20260806 12:24:03.112685 9775 consensus_queue.cc:237] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.113979 9775 tablet_replica.cc:439] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: TxnStatusTablet state changed. Reason: New leader 16225fa0e41d46ad91f39b67765ce083. Latest consensus state: current_term: 1 leader_uuid: "16225fa0e41d46ad91f39b67765ce083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } }
I20260806 12:24:03.114085 9775 tablet_replica.cc:442] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: This TxnStatusTablet replica's current role is: LEADER
I20260806 12:24:03.114320 9813 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 12:24:03.114461 9813 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 12:24:03.114641 8968 catalog_manager.cc:5495] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66). New cstate: current_term: 1 leader_uuid: "16225fa0e41d46ad91f39b67765ce083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } health_report { overall_health: UNKNOWN } } }
I20260806 12:24:03.137969 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:03.133796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.28.210.66:37315)
0806 12:24:03.133947 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:03.133951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:03.134115 (+ 164us) server_negotiation.cc:407] Connection header received
0806 12:24:03.134218 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:03.134220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:03.134283 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:03.134357 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:24:03.134676 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.135233 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.136591 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.136851 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.136954 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:03.137396 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:24:03.137496 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:24:03.137611 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:03.137780 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 12:24:03.137846 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 12:24:03.137991 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:03.133658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:37315 (local address 127.0.0.1:56162)
0806 12:24:03.133921 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:03.133939 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:03.134073 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:03.134389 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:03.134392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:03.134399 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 12:24:03.134528 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:03.134534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.135354 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.135358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:03.136401 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:03.136414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.136975 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.136979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:03.137112 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:03.137185 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 12:24:03.137624 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 12:24:03.137642 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 12:24:03.137715 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 12:24:03.137824 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 12:24:03.141412 9214 raft_consensus.cc:1261] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Refusing update from remote peer 16225fa0e41d46ad91f39b67765ce083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:03.141412 9637 raft_consensus.cc:1261] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Refusing update from remote peer 16225fa0e41d46ad91f39b67765ce083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:03.142133 9760 consensus_queue.cc:1035] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:03.142323 9775 consensus_queue.cc:1035] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:03.143863 9763 tablet_replica.cc:439] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442: TxnStatusTablet state changed. Reason: New leader 16225fa0e41d46ad91f39b67765ce083. Latest consensus state: current_term: 1 leader_uuid: "16225fa0e41d46ad91f39b67765ce083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } }
I20260806 12:24:03.144028 9763 tablet_replica.cc:442] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:24:03.144618 9792 tablet_replica.cc:439] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2: TxnStatusTablet state changed. Reason: New leader 16225fa0e41d46ad91f39b67765ce083. Latest consensus state: current_term: 1 leader_uuid: "16225fa0e41d46ad91f39b67765ce083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } }
I20260806 12:24:03.144774 9792 tablet_replica.cc:442] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:24:03.146813 9760 tablet_replica.cc:439] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16225fa0e41d46ad91f39b67765ce083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } }
I20260806 12:24:03.146991 9760 tablet_replica.cc:442] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: This TxnStatusTablet replica's current role is: LEADER
I20260806 12:24:03.149170 9792 tablet_replica.cc:439] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16225fa0e41d46ad91f39b67765ce083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } }
I20260806 12:24:03.149310 9792 tablet_replica.cc:442] T a7347a5bdb3e42cdbd0e6a16feae297d P 42a2f4f8f89849cd909d8faa51264fb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:24:03.150390 9770 mvcc.cc:205] Tried to move back new op lower bound from 7315534000701927424 to 7315534000590987264. Current Snapshot: MvccSnapshot[applied={T|T < 7315534000701927424 or (T in {7315534000701927424})}]
I20260806 12:24:03.151521 9766 tablet_replica.cc:439] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16225fa0e41d46ad91f39b67765ce083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } }
I20260806 12:24:03.151690 9766 tablet_replica.cc:442] T a7347a5bdb3e42cdbd0e6a16feae297d P 16225fa0e41d46ad91f39b67765ce083: This TxnStatusTablet replica's current role is: LEADER
I20260806 12:24:03.153272 9763 tablet_replica.cc:439] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16225fa0e41d46ad91f39b67765ce083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } } }
I20260806 12:24:03.153426 9763 tablet_replica.cc:442] T a7347a5bdb3e42cdbd0e6a16feae297d P df5a32b07c884f19bbf9b26799b48442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 12:24:03.159199 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:03.154980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.28.210.67:34261)
0806 12:24:03.155140 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:03.155146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:03.155162 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:24:03.155257 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:03.155261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:03.155329 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:03.155467 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:24:03.155746 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.156577 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.157415 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.157767 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.157865 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:03.158118 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:24:03.158756 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:24:03.158867 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:03.158978 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 12:24:03.159045 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 12:24:03.171677 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:03.167857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:37315 (local address 127.28.210.67:60503)
0806 12:24:03.168326 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:03.168339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:03.168437 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:03.168685 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:03.168687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:03.168692 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 12:24:03.168796 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:03.168800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.169793 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.169796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:03.170420 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:03.170429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.170998 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.171002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:03.171082 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:03.171135 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 12:24:03.171423 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 12:24:03.171427 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 12:24:03.171468 (+ 41us) client_negotiation.cc:240] Negotiation successful
0806 12:24:03.171536 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":154,"threads_started":1}
I20260806 12:24:03.171680 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:03.168005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:60503 (local address 127.28.210.66:37315)
0806 12:24:03.168178 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:03.168183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:03.168402 (+ 219us) server_negotiation.cc:407] Connection header received
0806 12:24:03.168530 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:03.168533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:03.168591 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:03.168681 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:24:03.168880 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.169665 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.170526 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.170865 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.170973 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:03.171263 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:24:03.171350 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:24:03.171410 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:03.171482 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 12:24:03.171541 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 12:24:03.179574 9214 raft_consensus.cc:1261] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Refusing update from remote peer 42a2f4f8f89849cd909d8faa51264fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:03.179765 9425 raft_consensus.cc:1261] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Refusing update from remote peer 42a2f4f8f89849cd909d8faa51264fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:03.180119 9762 consensus_queue.cc:1035] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:03.180315 9792 consensus_queue.cc:1035] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:03.207361 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:03.203310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:33859 (local address 127.28.210.67:34261)
0806 12:24:03.203429 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:03.203434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:03.203718 (+ 284us) server_negotiation.cc:407] Connection header received
0806 12:24:03.203891 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:03.203894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:03.203946 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:03.204016 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:24:03.204337 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.204859 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.206089 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.206359 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.206453 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:03.206827 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:24:03.206928 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:24:03.207012 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:03.207145 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 12:24:03.207211 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 12:24:03.207809 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:03.203233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:34261 (local address 127.28.210.66:33859)
0806 12:24:03.203633 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:03.203649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:03.203761 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:03.204034 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:03.204037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:03.204045 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 12:24:03.204209 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:03.204215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.204969 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.204972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:03.205933 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:03.205944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.206469 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.206472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:03.206580 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:03.206639 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 12:24:03.207028 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 12:24:03.207035 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 12:24:03.207576 (+ 541us) client_negotiation.cc:240] Negotiation successful
0806 12:24:03.207668 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
I20260806 12:24:03.240662 8969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:24:03.240913 8969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:24:03.243985 8969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa: Sending DeleteTablet for 3 replicas of tablet 5c9a154dfab64d8db791f8fba40bc441
I20260806 12:24:03.244854 9183 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9a154dfab64d8db791f8fba40bc441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:03 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:49922
I20260806 12:24:03.245095 9782 tablet_replica.cc:330] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442: stopping tablet replica
I20260806 12:24:03.245229 9782 raft_consensus.cc:2229] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:24:03.245390 9782 raft_consensus.cc:2258] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:03.246407 9782 ts_tablet_manager.cc:1905] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:03.248185 9402 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9a154dfab64d8db791f8fba40bc441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:03 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:56152
I20260806 12:24:03.248232 9606 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9a154dfab64d8db791f8fba40bc441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:03 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:59304
I20260806 12:24:03.248422 9783 tablet_replica.cc:330] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083: stopping tablet replica
I20260806 12:24:03.248448 9785 tablet_replica.cc:330] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2: stopping tablet replica
I20260806 12:24:03.248556 9783 raft_consensus.cc:2229] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:24:03.248556 9785 raft_consensus.cc:2229] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:24:03.248692 9783 raft_consensus.cc:2258] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:03.248943 9785 raft_consensus.cc:2258] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:03.250669 9783 ts_tablet_manager.cc:1905] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:03.254972 9783 ts_tablet_manager.cc:1918] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:24:03.255136 9783 log.cc:1196] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/5c9a154dfab64d8db791f8fba40bc441
I20260806 12:24:03.255347 9782 ts_tablet_manager.cc:1918] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:24:03.255447 9783 ts_tablet_manager.cc:1939] T 5c9a154dfab64d8db791f8fba40bc441 P 16225fa0e41d46ad91f39b67765ce083: Deleting consensus metadata
I20260806 12:24:03.255530 9782 log.cc:1196] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/wal/wals/5c9a154dfab64d8db791f8fba40bc441
I20260806 12:24:03.255813 9782 ts_tablet_manager.cc:1939] T 5c9a154dfab64d8db791f8fba40bc441 P df5a32b07c884f19bbf9b26799b48442: Deleting consensus metadata
I20260806 12:24:03.257082 9785 ts_tablet_manager.cc:1905] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:03.259332 8947 catalog_manager.cc:4841] TS 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66:37315): tablet 5c9a154dfab64d8db791f8fba40bc441 (table test-table [id=e619e7da00bd48e4beabb2e25b387dd8]) successfully deleted
I20260806 12:24:03.259797 8950 catalog_manager.cc:4841] TS df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455): tablet 5c9a154dfab64d8db791f8fba40bc441 (table test-table [id=e619e7da00bd48e4beabb2e25b387dd8]) successfully deleted
I20260806 12:24:03.261301 9785 ts_tablet_manager.cc:1918] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:24:03.261415 9785 log.cc:1196] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/5c9a154dfab64d8db791f8fba40bc441
I20260806 12:24:03.261705 9785 ts_tablet_manager.cc:1939] T 5c9a154dfab64d8db791f8fba40bc441 P 42a2f4f8f89849cd909d8faa51264fb2: Deleting consensus metadata
I20260806 12:24:03.263576 8951 catalog_manager.cc:4841] TS 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67:34261): tablet 5c9a154dfab64d8db791f8fba40bc441 (table test-table [id=e619e7da00bd48e4beabb2e25b387dd8]) successfully deleted
Aug 06 12:24:03 dist-test-slave-lgr5 krb5kdc[8897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 12:24:03 dist-test-slave-lgr5 krb5kdc[8897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:03 dist-test-slave-lgr5 krb5kdc[8897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 12:24:03.296038 9043 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:03.287837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.28.210.126:40067)
0806 12:24:03.288006 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:03.288011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:03.288261 (+ 250us) server_negotiation.cc:407] Connection header received
0806 12:24:03.288428 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:03.288432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:03.288498 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:03.288609 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:03.289351 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.290190 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.291206 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.291591 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.291707 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:03.295856 (+ 4149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52496: BlockingRecv error: recv got EOF from 127.0.0.1:52496 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 12:24:03 dist-test-slave-lgr5 krb5kdc[8897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM
I20260806 12:24:03.310005 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:03.297890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52508 (local address 127.28.210.126:40067)
0806 12:24:03.298031 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:03.298036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:03.299843 (+ 1807us) server_negotiation.cc:407] Connection header received
0806 12:24:03.300015 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:03.300019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:03.300070 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:03.300171 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:03.300928 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.301493 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.302373 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.302608 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.302702 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:03.305724 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:03.305730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:03.305733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:03.305781 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:03.307688 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:03.308249 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:03.308253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:03.308256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:03.308372 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:03.308898 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:03.308901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:03.308903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:03.309180 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:03.309358 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:03.309660 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 12:24:03.309806 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 12:24:03.313234 8969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:24:03.313767 8969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:24:03.319192 9183 tablet_service.cc:1467] Processing CreateTablet for tablet 1a199694b3c64296b9bd8a4dccfee38e (DEFAULT_TABLE table=test-table [id=3b006d2e58494316aba651d00fc49839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:03.319265 9606 tablet_service.cc:1467] Processing CreateTablet for tablet 1a199694b3c64296b9bd8a4dccfee38e (DEFAULT_TABLE table=test-table [id=3b006d2e58494316aba651d00fc49839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:03.319237 9402 tablet_service.cc:1467] Processing CreateTablet for tablet 1a199694b3c64296b9bd8a4dccfee38e (DEFAULT_TABLE table=test-table [id=3b006d2e58494316aba651d00fc49839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:03.319581 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a199694b3c64296b9bd8a4dccfee38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:03.319581 9402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a199694b3c64296b9bd8a4dccfee38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:03.319583 9183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a199694b3c64296b9bd8a4dccfee38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:03.322228 9753 tablet_bootstrap.cc:492] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083: Bootstrap starting.
I20260806 12:24:03.322283 9750 tablet_bootstrap.cc:492] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442: Bootstrap starting.
I20260806 12:24:03.322569 9755 tablet_bootstrap.cc:492] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2: Bootstrap starting.
I20260806 12:24:03.323290 9753 tablet_bootstrap.cc:654] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:03.323405 9750 tablet_bootstrap.cc:654] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:03.323525 9755 tablet_bootstrap.cc:654] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:03.325503 9750 tablet_bootstrap.cc:492] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442: No bootstrap required, opened a new log
I20260806 12:24:03.325620 9750 ts_tablet_manager.cc:1397] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:24:03.325706 9755 tablet_bootstrap.cc:492] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2: No bootstrap required, opened a new log
I20260806 12:24:03.325706 9753 tablet_bootstrap.cc:492] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083: No bootstrap required, opened a new log
I20260806 12:24:03.325794 9755 ts_tablet_manager.cc:1397] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:24:03.325794 9753 ts_tablet_manager.cc:1397] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 12:24:03.326227 9750 raft_consensus.cc:348] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.326274 9753 raft_consensus.cc:348] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.326385 9750 raft_consensus.cc:374] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:03.326422 9753 raft_consensus.cc:374] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:03.326422 9750 raft_consensus.cc:729] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5a32b07c884f19bbf9b26799b48442, State: Initialized, Role: FOLLOWER
I20260806 12:24:03.326459 9753 raft_consensus.cc:729] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16225fa0e41d46ad91f39b67765ce083, State: Initialized, Role: FOLLOWER
I20260806 12:24:03.326465 9755 raft_consensus.cc:348] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.326570 9755 raft_consensus.cc:374] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:03.326603 9755 raft_consensus.cc:729] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a2f4f8f89849cd909d8faa51264fb2, State: Initialized, Role: FOLLOWER
I20260806 12:24:03.326591 9753 consensus_queue.cc:260] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.326699 9750 consensus_queue.cc:260] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.326723 9755 consensus_queue.cc:260] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.326876 9753 ts_tablet_manager.cc:1428] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 12:24:03.326967 9755 ts_tablet_manager.cc:1428] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 12:24:03.328588 9750 ts_tablet_manager.cc:1428] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:24:03.573170 9792 raft_consensus.cc:482] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:24:03.573328 9792 raft_consensus.cc:504] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.573791 9792 leader_election.cc:288] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455), 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66:37315)
I20260806 12:24:03.574301 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a199694b3c64296b9bd8a4dccfee38e" candidate_uuid: "42a2f4f8f89849cd909d8faa51264fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a32b07c884f19bbf9b26799b48442" is_pre_election: true
I20260806 12:24:03.574306 9425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a199694b3c64296b9bd8a4dccfee38e" candidate_uuid: "42a2f4f8f89849cd909d8faa51264fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16225fa0e41d46ad91f39b67765ce083" is_pre_election: true
I20260806 12:24:03.574465 9214 raft_consensus.cc:2454] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42a2f4f8f89849cd909d8faa51264fb2 in term 0.
I20260806 12:24:03.574469 9425 raft_consensus.cc:2454] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42a2f4f8f89849cd909d8faa51264fb2 in term 0.
I20260806 12:24:03.574803 9535 leader_election.cc:302] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a2f4f8f89849cd909d8faa51264fb2, df5a32b07c884f19bbf9b26799b48442; no voters:
I20260806 12:24:03.575003 9792 raft_consensus.cc:2781] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:24:03.575080 9792 raft_consensus.cc:482] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:24:03.575110 9792 raft_consensus.cc:3037] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:03.576179 9792 raft_consensus.cc:504] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.576561 9792 leader_election.cc:288] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [CANDIDATE]: Term 1 election: Requested vote from peers df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455), 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66:37315)
I20260806 12:24:03.576963 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a199694b3c64296b9bd8a4dccfee38e" candidate_uuid: "42a2f4f8f89849cd909d8faa51264fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a32b07c884f19bbf9b26799b48442"
I20260806 12:24:03.577090 9214 raft_consensus.cc:3037] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:03.577071 9425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a199694b3c64296b9bd8a4dccfee38e" candidate_uuid: "42a2f4f8f89849cd909d8faa51264fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16225fa0e41d46ad91f39b67765ce083"
I20260806 12:24:03.577234 9425 raft_consensus.cc:3037] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:03.578146 9214 raft_consensus.cc:2454] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42a2f4f8f89849cd909d8faa51264fb2 in term 1.
I20260806 12:24:03.578368 9425 raft_consensus.cc:2454] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42a2f4f8f89849cd909d8faa51264fb2 in term 1.
I20260806 12:24:03.578501 9535 leader_election.cc:302] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a2f4f8f89849cd909d8faa51264fb2, df5a32b07c884f19bbf9b26799b48442; no voters:
I20260806 12:24:03.578681 9792 raft_consensus.cc:2781] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:24:03.578765 9792 raft_consensus.cc:686] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 42a2f4f8f89849cd909d8faa51264fb2, State: Running, Role: LEADER
I20260806 12:24:03.578912 9792 consensus_queue.cc:237] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.580925 8968 catalog_manager.cc:5495] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67). New cstate: current_term: 1 leader_uuid: "42a2f4f8f89849cd909d8faa51264fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } health_report { overall_health: HEALTHY } } }
I20260806 12:24:03.636471 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:03.633239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.28.210.67:34261)
0806 12:24:03.633397 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:03.633402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:03.633418 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:24:03.633543 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:03.633547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:03.633600 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:03.633690 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:24:03.633946 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.634455 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.635350 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:03.635568 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:03.635663 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:03.635948 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:24:03.636040 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:24:03.636137 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:03.636238 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 12:24:03.636313 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 12:24:03.639394 9214 raft_consensus.cc:1261] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Refusing update from remote peer 42a2f4f8f89849cd909d8faa51264fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:03.639394 9425 raft_consensus.cc:1261] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Refusing update from remote peer 42a2f4f8f89849cd909d8faa51264fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:03.639952 9826 consensus_queue.cc:1035] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:03.640194 9792 consensus_queue.cc:1035] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:03.650941 8972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:24:03.651166 8972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:24:03.655122 8972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa: Sending DeleteTablet for 3 replicas of tablet 1a199694b3c64296b9bd8a4dccfee38e
I20260806 12:24:03.655915 9183 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a199694b3c64296b9bd8a4dccfee38e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:03 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:49922
I20260806 12:24:03.656155 9782 tablet_replica.cc:330] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442: stopping tablet replica
I20260806 12:24:03.656304 9782 raft_consensus.cc:2229] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:24:03.656275 9606 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a199694b3c64296b9bd8a4dccfee38e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:03 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:59304
I20260806 12:24:03.656456 9782 raft_consensus.cc:2258] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:03.656476 9785 tablet_replica.cc:330] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2: stopping tablet replica
I20260806 12:24:03.656610 9785 raft_consensus.cc:2229] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:24:03.656867 9785 raft_consensus.cc:2258] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:03.657191 9782 ts_tablet_manager.cc:1905] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:03.657707 9785 ts_tablet_manager.cc:1905] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:03.659479 9782 ts_tablet_manager.cc:1918] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:24:03.659587 9782 log.cc:1196] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-0/wal/wals/1a199694b3c64296b9bd8a4dccfee38e
I20260806 12:24:03.659816 9782 ts_tablet_manager.cc:1939] T 1a199694b3c64296b9bd8a4dccfee38e P df5a32b07c884f19bbf9b26799b48442: Deleting consensus metadata
I20260806 12:24:03.660734 9606 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a199694b3c64296b9bd8a4dccfee38e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:03 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:59304
I20260806 12:24:03.660763 9402 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a199694b3c64296b9bd8a4dccfee38e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:03 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:56152
I20260806 12:24:03.660851 9183 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a199694b3c64296b9bd8a4dccfee38e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:03 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:49922
I20260806 12:24:03.660964 9783 tablet_replica.cc:330] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083: stopping tablet replica
I20260806 12:24:03.661113 9783 raft_consensus.cc:2229] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:24:03.661271 9783 raft_consensus.cc:2258] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:03.661298 9785 ts_tablet_manager.cc:1918] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:24:03.661402 9785 log.cc:1196] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/1a199694b3c64296b9bd8a4dccfee38e
I20260806 12:24:03.661721 9785 ts_tablet_manager.cc:1939] T 1a199694b3c64296b9bd8a4dccfee38e P 42a2f4f8f89849cd909d8faa51264fb2: Deleting consensus metadata
I20260806 12:24:03.662158 9783 ts_tablet_manager.cc:1905] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:03.662711 8951 catalog_manager.cc:4841] TS 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67:34261): tablet 1a199694b3c64296b9bd8a4dccfee38e (table test-table [id=3b006d2e58494316aba651d00fc49839]) successfully deleted
W20260806 12:24:03.663070 8951 catalog_manager.cc:4808] TS 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67:34261): delete failed for tablet 1a199694b3c64296b9bd8a4dccfee38e because the tablet was not found. No further retry: Not found: Tablet not found: 1a199694b3c64296b9bd8a4dccfee38e
I20260806 12:24:03.662724 8950 catalog_manager.cc:4841] TS df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455): tablet 1a199694b3c64296b9bd8a4dccfee38e (table test-table [id=3b006d2e58494316aba651d00fc49839]) successfully deleted
W20260806 12:24:03.663439 8950 catalog_manager.cc:4808] TS df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455): delete failed for tablet 1a199694b3c64296b9bd8a4dccfee38e because the tablet was not found. No further retry: Not found: Tablet not found: 1a199694b3c64296b9bd8a4dccfee38e
I20260806 12:24:03.664040 8972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:24:03.664739 8972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:24:03.664901 9783 ts_tablet_manager.cc:1918] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:24:03.665006 9783 log.cc:1196] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/1a199694b3c64296b9bd8a4dccfee38e
I20260806 12:24:03.665242 9783 ts_tablet_manager.cc:1939] T 1a199694b3c64296b9bd8a4dccfee38e P 16225fa0e41d46ad91f39b67765ce083: Deleting consensus metadata
I20260806 12:24:03.666046 8947 catalog_manager.cc:4841] TS 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66:37315): tablet 1a199694b3c64296b9bd8a4dccfee38e (table test-table [id=3b006d2e58494316aba651d00fc49839]) successfully deleted
I20260806 12:24:03.670359 9183 tablet_service.cc:1467] Processing CreateTablet for tablet 2b04f9a553b145a5aaeab930a06c7a19 (DEFAULT_TABLE table=test-table [id=f83204292184451d9d16779908efd5c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:03.670455 9402 tablet_service.cc:1467] Processing CreateTablet for tablet 2b04f9a553b145a5aaeab930a06c7a19 (DEFAULT_TABLE table=test-table [id=f83204292184451d9d16779908efd5c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:03.670537 9606 tablet_service.cc:1467] Processing CreateTablet for tablet 2b04f9a553b145a5aaeab930a06c7a19 (DEFAULT_TABLE table=test-table [id=f83204292184451d9d16779908efd5c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:03.670742 9402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b04f9a553b145a5aaeab930a06c7a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:03.670745 9183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b04f9a553b145a5aaeab930a06c7a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:03.670781 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b04f9a553b145a5aaeab930a06c7a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:03.672986 9753 tablet_bootstrap.cc:492] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083: Bootstrap starting.
I20260806 12:24:03.673290 9750 tablet_bootstrap.cc:492] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442: Bootstrap starting.
I20260806 12:24:03.673331 9755 tablet_bootstrap.cc:492] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2: Bootstrap starting.
I20260806 12:24:03.673947 9753 tablet_bootstrap.cc:654] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:03.674115 9755 tablet_bootstrap.cc:654] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:03.674253 9750 tablet_bootstrap.cc:654] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:03.675251 9753 tablet_bootstrap.cc:492] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083: No bootstrap required, opened a new log
I20260806 12:24:03.675357 9753 ts_tablet_manager.cc:1397] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 12:24:03.675788 9753 raft_consensus.cc:348] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.675936 9753 raft_consensus.cc:374] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:03.675963 9753 raft_consensus.cc:729] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16225fa0e41d46ad91f39b67765ce083, State: Initialized, Role: FOLLOWER
I20260806 12:24:03.676107 9753 consensus_queue.cc:260] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.676385 9753 ts_tablet_manager.cc:1428] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 12:24:03.677141 9755 tablet_bootstrap.cc:492] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2: No bootstrap required, opened a new log
I20260806 12:24:03.677243 9755 ts_tablet_manager.cc:1397] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 12:24:03.677500 9750 tablet_bootstrap.cc:492] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442: No bootstrap required, opened a new log
I20260806 12:24:03.677577 9750 ts_tablet_manager.cc:1397] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 12:24:03.677711 9755 raft_consensus.cc:348] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.677839 9755 raft_consensus.cc:374] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:03.677874 9755 raft_consensus.cc:729] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a2f4f8f89849cd909d8faa51264fb2, State: Initialized, Role: FOLLOWER
I20260806 12:24:03.678023 9755 consensus_queue.cc:260] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.678082 9750 raft_consensus.cc:348] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.678216 9750 raft_consensus.cc:374] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:03.678256 9750 raft_consensus.cc:729] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5a32b07c884f19bbf9b26799b48442, State: Initialized, Role: FOLLOWER
I20260806 12:24:03.678325 9755 ts_tablet_manager.cc:1428] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 12:24:03.678429 9750 consensus_queue.cc:260] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.678773 9750 ts_tablet_manager.cc:1428] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 12:24:03.726670 9762 raft_consensus.cc:482] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:24:03.726818 9762 raft_consensus.cc:504] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.727308 9762 leader_election.cc:288] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455), 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66:37315)
I20260806 12:24:03.727830 9425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b04f9a553b145a5aaeab930a06c7a19" candidate_uuid: "42a2f4f8f89849cd909d8faa51264fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16225fa0e41d46ad91f39b67765ce083" is_pre_election: true
I20260806 12:24:03.727859 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b04f9a553b145a5aaeab930a06c7a19" candidate_uuid: "42a2f4f8f89849cd909d8faa51264fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a32b07c884f19bbf9b26799b48442" is_pre_election: true
I20260806 12:24:03.727990 9425 raft_consensus.cc:2454] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42a2f4f8f89849cd909d8faa51264fb2 in term 0.
I20260806 12:24:03.728010 9214 raft_consensus.cc:2454] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42a2f4f8f89849cd909d8faa51264fb2 in term 0.
I20260806 12:24:03.728365 9533 leader_election.cc:302] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16225fa0e41d46ad91f39b67765ce083, 42a2f4f8f89849cd909d8faa51264fb2, df5a32b07c884f19bbf9b26799b48442; no voters:
I20260806 12:24:03.728710 9762 raft_consensus.cc:2781] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:24:03.728782 9762 raft_consensus.cc:482] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:24:03.728822 9762 raft_consensus.cc:3037] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:03.730050 9762 raft_consensus.cc:504] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.730429 9762 leader_election.cc:288] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [CANDIDATE]: Term 1 election: Requested vote from peers df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455), 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66:37315)
I20260806 12:24:03.730856 9425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b04f9a553b145a5aaeab930a06c7a19" candidate_uuid: "42a2f4f8f89849cd909d8faa51264fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16225fa0e41d46ad91f39b67765ce083"
I20260806 12:24:03.730866 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b04f9a553b145a5aaeab930a06c7a19" candidate_uuid: "42a2f4f8f89849cd909d8faa51264fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a32b07c884f19bbf9b26799b48442"
I20260806 12:24:03.730988 9425 raft_consensus.cc:3037] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:03.731000 9214 raft_consensus.cc:3037] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:03.731943 9214 raft_consensus.cc:2454] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42a2f4f8f89849cd909d8faa51264fb2 in term 1.
I20260806 12:24:03.731954 9425 raft_consensus.cc:2454] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42a2f4f8f89849cd909d8faa51264fb2 in term 1.
I20260806 12:24:03.732273 9535 leader_election.cc:302] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a2f4f8f89849cd909d8faa51264fb2, df5a32b07c884f19bbf9b26799b48442; no voters:
I20260806 12:24:03.732446 9762 raft_consensus.cc:2781] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:24:03.732539 9762 raft_consensus.cc:686] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 42a2f4f8f89849cd909d8faa51264fb2, State: Running, Role: LEADER
I20260806 12:24:03.732686 9762 consensus_queue.cc:237] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } }
I20260806 12:24:03.734426 8968 catalog_manager.cc:5495] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67). New cstate: current_term: 1 leader_uuid: "42a2f4f8f89849cd909d8faa51264fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42a2f4f8f89849cd909d8faa51264fb2" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34261 } health_report { overall_health: HEALTHY } } }
I20260806 12:24:03.758898 9425 raft_consensus.cc:1261] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Refusing update from remote peer 42a2f4f8f89849cd909d8faa51264fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:03.758896 9214 raft_consensus.cc:1261] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Refusing update from remote peer 42a2f4f8f89849cd909d8faa51264fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:03.759495 9762 consensus_queue.cc:1035] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5a32b07c884f19bbf9b26799b48442" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:03.759692 9792 consensus_queue.cc:1035] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16225fa0e41d46ad91f39b67765ce083" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:03.801906 8968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:24:03.802090 8968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:24:03.804450 8968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28fed4920a9346c6abf99c9e4a50d3fa: Sending DeleteTablet for 3 replicas of tablet 2b04f9a553b145a5aaeab930a06c7a19
I20260806 12:24:03.805107 9183 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b04f9a553b145a5aaeab930a06c7a19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:03 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:49922
I20260806 12:24:03.805130 9402 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b04f9a553b145a5aaeab930a06c7a19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:03 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:56152
I20260806 12:24:03.805223 9606 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b04f9a553b145a5aaeab930a06c7a19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:03 UTC) from {username='oryx', principal='oryx/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:59304
I20260806 12:24:03.805327 9782 tablet_replica.cc:330] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442: stopping tablet replica
I20260806 12:24:03.805420 9785 tablet_replica.cc:330] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2: stopping tablet replica
I20260806 12:24:03.805444 9782 raft_consensus.cc:2229] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:24:03.805454 9783 tablet_replica.cc:330] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083: stopping tablet replica
I20260806 12:24:03.805531 9785 raft_consensus.cc:2229] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:24:03.805569 9782 raft_consensus.cc:2258] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:03.805569 9783 raft_consensus.cc:2229] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:24:03.805670 9783 raft_consensus.cc:2258] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:03.805707 9785 raft_consensus.cc:2258] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:03.806351 9785 ts_tablet_manager.cc:1905] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:03.806416 9782 ts_tablet_manager.cc:1905] T 2b04f9a553b145a5aaeab930a06c7a19 P df5a32b07c884f19bbf9b26799b48442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:03.806579 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 9085
I20260806 12:24:03.809229 9783 ts_tablet_manager.cc:1905] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:03.809623 9785 ts_tablet_manager.cc:1918] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:24:03.809696 9785 log.cc:1196] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/2b04f9a553b145a5aaeab930a06c7a19
I20260806 12:24:03.809949 9785 ts_tablet_manager.cc:1939] T 2b04f9a553b145a5aaeab930a06c7a19 P 42a2f4f8f89849cd909d8faa51264fb2: Deleting consensus metadata
I20260806 12:24:03.810750 8951 catalog_manager.cc:4841] TS 42a2f4f8f89849cd909d8faa51264fb2 (127.28.210.67:34261): tablet 2b04f9a553b145a5aaeab930a06c7a19 (table test-table [id=f83204292184451d9d16779908efd5c1]) successfully deleted
I20260806 12:24:03.811507 9783 ts_tablet_manager.cc:1918] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 12:24:03.811583 9783 log.cc:1196] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/2b04f9a553b145a5aaeab930a06c7a19
I20260806 12:24:03.811903 9783 ts_tablet_manager.cc:1939] T 2b04f9a553b145a5aaeab930a06c7a19 P 16225fa0e41d46ad91f39b67765ce083: Deleting consensus metadata
I20260806 12:24:03.813722 8947 catalog_manager.cc:4841] TS 16225fa0e41d46ad91f39b67765ce083 (127.28.210.66:37315): tablet 2b04f9a553b145a5aaeab930a06c7a19 (table test-table [id=f83204292184451d9d16779908efd5c1]) successfully deleted
I20260806 12:24:03.822418 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 9304
W20260806 12:24:03.822398 8950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:34455 (error 108)
I20260806 12:24:03.836982 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:03.835736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:34455 (local address 127.0.0.1:49936)
0806 12:24:03.836703 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:03.836791 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:34455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":489,"threads_started":1}
W20260806 12:24:03.837203 8950 catalog_manager.cc:4584] TS df5a32b07c884f19bbf9b26799b48442 (127.28.210.65:34455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b04f9a553b145a5aaeab930a06c7a19: Network error: Client connection negotiation failed: client connection to 127.28.210.65:34455: connect: Connection refused (error 111)
I20260806 12:24:03.844841 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 9514
I20260806 12:24:03.868343 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 8926
2026-08-06T12:24:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3643 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:24:03 dist-test-slave-lgr5 krb5kdc[9883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:24:03 dist-test-slave-lgr5 krb5kdc[9883](info): set up 2 sockets
Aug 06 12:24:03 dist-test-slave-lgr5 krb5kdc[9883](info): commencing operation
krb5kdc: starting...
W20260806 12:24:05.055460 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.008s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:24:05 dist-test-slave-lgr5 krb5kdc[9883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:24:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:24:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:24:05.228266 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:42229
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:42229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:05.356385 9902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:05.356755 9902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:05.356822 9902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:05.361280 9902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:24:05.361375 9902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:05.361403 9902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:05.361425 9902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:24:05.361446 9902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:24:05.367357 9902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:42229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:42229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:05.368673 9902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:05.369804 9902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:05.379551 9910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:05.379675 9902 server_base.cc:1034] running on GCE node
W20260806 12:24:05.379567 9916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:05.379570 9908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:05.380846 9902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:05.382206 9902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:05.383428 9902 hybrid_clock.cc:648] HybridClock initialized: now 1786019045383408 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T12:24:05Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1516 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:24:05 dist-test-slave-lgr5 krb5kdc[9925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:24:05 dist-test-slave-lgr5 krb5kdc[9925](info): set up 2 sockets
Aug 06 12:24:05 dist-test-slave-lgr5 krb5kdc[9925](info): commencing operation
krb5kdc: starting...
W20260806 12:24:06.617918 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:24:06 dist-test-slave-lgr5 krb5kdc[9925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:24:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:24:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:24:06.772058 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34519
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:34519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:06.895509 9944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:06.895826 9944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:06.895869 9944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:06.899919 9944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:24:06.899991 9944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:06.900010 9944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:06.900027 9944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:24:06.900041 9944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:24:06.905709 9944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:34519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:06.906919 9944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:06.907971 9944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:06.917205 9953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:06.917277 9950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:06.917205 9951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:06.917387 9944 server_base.cc:1034] running on GCE node
I20260806 12:24:06.918188 9944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:06.919637 9944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:06.920881 9944 hybrid_clock.cc:648] HybridClock initialized: now 1786019046920841 us; error 55 us; skew 500 ppm
Aug 06 12:24:06 dist-test-slave-lgr5 krb5kdc[9925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019046, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:06.924299 9944 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:24:06.925503 9944 webserver.cc:460] Webserver started at http://127.28.210.126:46811/ using document root <none> and password file <none>
I20260806 12:24:06.926115 9944 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:06.926187 9944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:06.926422 9944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:06.928829 9944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "9024cf00daa6439198061ff0bdadc91f"
format_stamp: "Formatted at 2026-08-06 12:24:06 on dist-test-slave-lgr5"
server_key: "28f4a212cc5ee3f9023938a15be4fb04"
server_key_iv: "a7805e342815b3a90d0fac9ecf862fd2"
server_key_version: "encryptionkey@0"
I20260806 12:24:06.929508 9944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "9024cf00daa6439198061ff0bdadc91f"
format_stamp: "Formatted at 2026-08-06 12:24:06 on dist-test-slave-lgr5"
server_key: "28f4a212cc5ee3f9023938a15be4fb04"
server_key_iv: "a7805e342815b3a90d0fac9ecf862fd2"
server_key_version: "encryptionkey@0"
I20260806 12:24:06.934173 9944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:24:06.937186 9963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:06.938637 9944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:24:06.938793 9944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "9024cf00daa6439198061ff0bdadc91f"
format_stamp: "Formatted at 2026-08-06 12:24:06 on dist-test-slave-lgr5"
server_key: "28f4a212cc5ee3f9023938a15be4fb04"
server_key_iv: "a7805e342815b3a90d0fac9ecf862fd2"
server_key_version: "encryptionkey@0"
I20260806 12:24:06.938920 9944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:06.975454 9944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:06.979377 9944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:06.979600 9944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:06.987208 9944 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:34519
I20260806 12:24:06.987277 10039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:34519 every 8 connection(s)
I20260806 12:24:06.988327 9944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:24:06.991530 10040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:06.991851 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 9944
I20260806 12:24:06.991981 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:24:06.992365 29513 external_mini_cluster.cc:1428] Setting key 02de8838e674c9d32813128b71ced12e
I20260806 12:24:06.999508 10040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f: Bootstrap starting.
Aug 06 12:24:06 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:24:07.003487 10040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:07.004525 10040 log.cc:824] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f: Log is configured to *not* fsync() on all Append() calls
I20260806 12:24:07.007362 10040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f: No bootstrap required, opened a new log
I20260806 12:24:07.009444 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:06.993905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.28.210.126:34519)
0806 12:24:06.994367 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:06.994384 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:06.994426 (+ 42us) server_negotiation.cc:407] Connection header received
0806 12:24:06.995184 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:06.995206 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:06.995560 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:06.995871 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:06.997046 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:06.998120 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:06.999315 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:06.999869 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.000010 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:07.003605 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:07.003614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:07.003636 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:07.003679 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:07.005663 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:07.006323 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:07.006330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:07.006336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:07.006520 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:07.006892 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:07.006896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:07.006897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:07.007476 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:07.007702 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:07.008024 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 12:24:07.008517 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":129,"threads_started":1}
I20260806 12:24:07.011853 10040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024cf00daa6439198061ff0bdadc91f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34519 } }
I20260806 12:24:07.012197 10040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:07.012260 10040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9024cf00daa6439198061ff0bdadc91f, State: Initialized, Role: FOLLOWER
I20260806 12:24:07.012728 10040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024cf00daa6439198061ff0bdadc91f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34519 } }
I20260806 12:24:07.012866 10040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:24:07.012926 10040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:24:07.013031 10040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:07.014765 10040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024cf00daa6439198061ff0bdadc91f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34519 } }
I20260806 12:24:07.015659 10040 leader_election.cc:302] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9024cf00daa6439198061ff0bdadc91f; no voters:
I20260806 12:24:07.016379 10040 leader_election.cc:288] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:24:07.016525 10045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:24:07.016866 10045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 1 LEADER]: Becoming Leader. State: Replica: 9024cf00daa6439198061ff0bdadc91f, State: Running, Role: LEADER
I20260806 12:24:07.017359 10045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024cf00daa6439198061ff0bdadc91f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34519 } }
I20260806 12:24:07.018785 10040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:24:07.019408 10047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9024cf00daa6439198061ff0bdadc91f. Latest consensus state: current_term: 1 leader_uuid: "9024cf00daa6439198061ff0bdadc91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024cf00daa6439198061ff0bdadc91f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34519 } } }
I20260806 12:24:07.019572 10047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [sys.catalog]: This master's current role is: LEADER
I20260806 12:24:07.019994 10051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:24:07.019950 10046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9024cf00daa6439198061ff0bdadc91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024cf00daa6439198061ff0bdadc91f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34519 } } }
I20260806 12:24:07.020131 10046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [sys.catalog]: This master's current role is: LEADER
I20260806 12:24:07.023252 10051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:24:07.030336 10051 catalog_manager.cc:1347] Generated new cluster ID: f27b4155ff9b42c1a40e322fe6a3b29d
I20260806 12:24:07.030452 10051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:24:07.050282 10051 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:24:07.051472 10051 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:24:07.061270 10051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f: Generated new TSK 0
I20260806 12:24:07.062141 10051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:24:07.107821 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34519
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:07.240374 10079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:07.240710 10079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:07.240757 10079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:07.245297 10079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:07.245421 10079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:07.245489 10079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:24:07.251926 10079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:07.253470 10079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:07.254805 10079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:07.266637 10091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:07.266637 10085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:07.266806 10087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:07.266845 10079 server_base.cc:1034] running on GCE node
I20260806 12:24:07.267490 10079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:07.268307 10079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:07.269506 10079 hybrid_clock.cc:648] HybridClock initialized: now 1786019047269482 us; error 40 us; skew 500 ppm
Aug 06 12:24:07 dist-test-slave-lgr5 krb5kdc[9925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:07.272882 10079 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:24:07.273990 10079 webserver.cc:460] Webserver started at http://127.28.210.65:46087/ using document root <none> and password file <none>
I20260806 12:24:07.274621 10079 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:07.274695 10079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:07.274922 10079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:07.277313 10079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "771bbb067a914c89b8ddfe9ee51ccf5f"
format_stamp: "Formatted at 2026-08-06 12:24:07 on dist-test-slave-lgr5"
server_key: "2d7ac9f20118012ce3021e1564d1f575"
server_key_iv: "1644db7e2c350fd55deb69f014f35cd4"
server_key_version: "encryptionkey@0"
I20260806 12:24:07.277956 10079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "771bbb067a914c89b8ddfe9ee51ccf5f"
format_stamp: "Formatted at 2026-08-06 12:24:07 on dist-test-slave-lgr5"
server_key: "2d7ac9f20118012ce3021e1564d1f575"
server_key_iv: "1644db7e2c350fd55deb69f014f35cd4"
server_key_version: "encryptionkey@0"
I20260806 12:24:07.282914 10079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 12:24:07.286362 10102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:07.287896 10079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:24:07.288053 10079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "771bbb067a914c89b8ddfe9ee51ccf5f"
format_stamp: "Formatted at 2026-08-06 12:24:07 on dist-test-slave-lgr5"
server_key: "2d7ac9f20118012ce3021e1564d1f575"
server_key_iv: "1644db7e2c350fd55deb69f014f35cd4"
server_key_version: "encryptionkey@0"
I20260806 12:24:07.288228 10079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:07.302397 10079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:07.305992 10079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:07.306206 10079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:07.306895 10079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:07.307869 10079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:07.307924 10079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:07.307982 10079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:07.308009 10079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:07.317530 10079 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:41217
I20260806 12:24:07.317574 10263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:41217 every 8 connection(s)
I20260806 12:24:07.318524 10079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:24:07.327754 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 10079
I20260806 12:24:07.327929 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:24:07.328368 29513 external_mini_cluster.cc:1428] Setting key 0750e3d82b322b06c928343f4efbdf5f
Aug 06 12:24:07 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:24:07.335119 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:07.320660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:45609 (local address 127.28.210.126:34519)
0806 12:24:07.320816 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:07.320820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:07.321864 (+ 1044us) server_negotiation.cc:407] Connection header received
0806 12:24:07.323251 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:07.323254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:07.323312 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:07.323396 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:07.324878 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:07.325455 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.326751 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:07.327017 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.327116 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:07.330682 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:07.330686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:07.330689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:07.330718 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:07.332466 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:07.333241 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:07.333245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:07.333247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:07.333374 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:07.333741 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:07.333745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:07.333746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:07.334031 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:07.334151 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:07.334740 (+ 589us) server_negotiation.cc:299] Negotiation successful
0806 12:24:07.334914 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 12:24:07.336135 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:07.321135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:45609)
0806 12:24:07.321678 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:07.321730 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:07.323004 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:07.323620 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:07.323628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:07.324046 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:07.324669 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:07.324688 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:07.325616 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.325620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:07.326604 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:07.326617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:07.327224 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.327228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:07.327483 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:07.328283 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:07.328304 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:07.330416 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:07.332685 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:07.332693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:07.332714 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:07.333106 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:07.333469 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:07.333470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:07.333472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:07.333643 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:07.334136 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:07.334142 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:07.334456 (+ 314us) client_negotiation.cc:769] Sending connection context
0806 12:24:07.334646 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 12:24:07.334992 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":131,"threads_started":1}
I20260806 12:24:07.338013 10265 heartbeater.cc:347] Connected to a master server at 127.28.210.126:34519
I20260806 12:24:07.338387 10265 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:07.339191 10265 heartbeater.cc:511] Master 127.28.210.126:34519 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:24:07.341378 9984 ts_manager.cc:194] Registered new tserver with Master: 771bbb067a914c89b8ddfe9ee51ccf5f (127.28.210.65:41217)
I20260806 12:24:07.342882 9984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:45609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:24:07.365444 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34519
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:07.491870 10270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:07.492254 10270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:07.492322 10270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:07.496665 10270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:07.496753 10270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:07.496848 10270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:24:07.502605 10270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:07.503892 10270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:07.505016 10270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:07.515196 10278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:07.515197 10281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:07.515205 10277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:07.515463 10270 server_base.cc:1034] running on GCE node
I20260806 12:24:07.516062 10270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:07.516815 10270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:07.518085 10270 hybrid_clock.cc:648] HybridClock initialized: now 1786019047517945 us; error 79 us; skew 500 ppm
Aug 06 12:24:07 dist-test-slave-lgr5 krb5kdc[9925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:07.521306 10270 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:24:07.522367 10270 webserver.cc:460] Webserver started at http://127.28.210.66:38155/ using document root <none> and password file <none>
I20260806 12:24:07.522918 10270 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:07.522993 10270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:07.523182 10270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:07.525377 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "dbeb51f14fec4829b7520ba8bcb27cd9"
format_stamp: "Formatted at 2026-08-06 12:24:07 on dist-test-slave-lgr5"
server_key: "4e630ec3cf2221e71ebe659a8652e98f"
server_key_iv: "85da24d4d798f3cd6798c644e5748f96"
server_key_version: "encryptionkey@0"
I20260806 12:24:07.525925 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "dbeb51f14fec4829b7520ba8bcb27cd9"
format_stamp: "Formatted at 2026-08-06 12:24:07 on dist-test-slave-lgr5"
server_key: "4e630ec3cf2221e71ebe659a8652e98f"
server_key_iv: "85da24d4d798f3cd6798c644e5748f96"
server_key_version: "encryptionkey@0"
I20260806 12:24:07.529860 10270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:24:07.532872 10295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:07.534301 10270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:24:07.534430 10270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "dbeb51f14fec4829b7520ba8bcb27cd9"
format_stamp: "Formatted at 2026-08-06 12:24:07 on dist-test-slave-lgr5"
server_key: "4e630ec3cf2221e71ebe659a8652e98f"
server_key_iv: "85da24d4d798f3cd6798c644e5748f96"
server_key_version: "encryptionkey@0"
I20260806 12:24:07.534569 10270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:07.555413 10270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:07.559299 10270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:07.559521 10270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:07.560334 10270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:07.561410 10270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:07.561471 10270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:07.561537 10270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:07.561570 10270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:07.573243 10270 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:42513
I20260806 12:24:07.573310 10448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:42513 every 8 connection(s)
I20260806 12:24:07.574373 10270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:24:07.584718 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 10270
I20260806 12:24:07.584836 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:24:07.585109 29513 external_mini_cluster.cc:1428] Setting key 644924e9e5080bcd34944fb0ac78c3a5
Aug 06 12:24:07 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:24:07.590335 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:07.576735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:50397 (local address 127.28.210.126:34519)
0806 12:24:07.576901 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:07.576904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:07.577972 (+ 1068us) server_negotiation.cc:407] Connection header received
0806 12:24:07.579493 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:07.579496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:07.579571 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:07.579645 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:07.581186 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:07.581857 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.583186 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:07.583436 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.583533 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:07.586284 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:07.586289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:07.586291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:07.586319 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:07.587833 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:07.588521 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:07.588526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:07.588527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:07.588619 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:07.589022 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:07.589026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:07.589027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:07.589250 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:07.589349 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:07.589967 (+ 618us) server_negotiation.cc:299] Negotiation successful
0806 12:24:07.590131 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 12:24:07.591346 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:07.577128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:50397)
0806 12:24:07.577731 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:07.577808 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:07.579198 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:07.579913 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:07.579922 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:07.580410 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:07.581003 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:07.581020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:07.582051 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.582054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:07.583041 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:07.583054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:07.583614 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.583619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:07.583809 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:07.584485 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:07.584503 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:07.586090 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:07.587987 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:07.587997 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:07.588022 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:07.588408 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:07.588712 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:07.588715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:07.588717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:07.588906 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:07.589365 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:07.589373 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:07.589689 (+ 316us) client_negotiation.cc:769] Sending connection context
0806 12:24:07.589884 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 12:24:07.590321 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":161,"threads_started":1}
I20260806 12:24:07.592770 10449 heartbeater.cc:347] Connected to a master server at 127.28.210.126:34519
I20260806 12:24:07.593060 10449 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:07.593652 10449 heartbeater.cc:511] Master 127.28.210.126:34519 requested a full tablet report, sending...
I20260806 12:24:07.594805 9984 ts_manager.cc:194] Registered new tserver with Master: dbeb51f14fec4829b7520ba8bcb27cd9 (127.28.210.66:42513)
I20260806 12:24:07.595438 9984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:50397
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:24:07.619752 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34519
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:07.748134 10454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:07.748456 10454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:07.748523 10454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:07.752817 10454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:07.752919 10454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:07.753012 10454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:24:07.759100 10454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:07.760466 10454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:07.761698 10454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:07.772964 10464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:07.772984 10463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:07.772984 10467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:07.773208 10454 server_base.cc:1034] running on GCE node
I20260806 12:24:07.773890 10454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:07.774659 10454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:07.775861 10454 hybrid_clock.cc:648] HybridClock initialized: now 1786019047775824 us; error 51 us; skew 500 ppm
Aug 06 12:24:07 dist-test-slave-lgr5 krb5kdc[9925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:07.778956 10454 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:24:07.780161 10454 webserver.cc:460] Webserver started at http://127.28.210.67:38901/ using document root <none> and password file <none>
I20260806 12:24:07.780812 10454 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:07.780889 10454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:07.781116 10454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:07.783614 10454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "45416e5017e041f09b50429dcac563ee"
format_stamp: "Formatted at 2026-08-06 12:24:07 on dist-test-slave-lgr5"
server_key: "ca2ae5280cb5671eca5c069d031de1fe"
server_key_iv: "7806e4471c6e235cd30629de16d479eb"
server_key_version: "encryptionkey@0"
I20260806 12:24:07.784387 10454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "45416e5017e041f09b50429dcac563ee"
format_stamp: "Formatted at 2026-08-06 12:24:07 on dist-test-slave-lgr5"
server_key: "ca2ae5280cb5671eca5c069d031de1fe"
server_key_iv: "7806e4471c6e235cd30629de16d479eb"
server_key_version: "encryptionkey@0"
I20260806 12:24:07.789220 10454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260806 12:24:07.792534 10476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:07.794199 10454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:24:07.794354 10454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "45416e5017e041f09b50429dcac563ee"
format_stamp: "Formatted at 2026-08-06 12:24:07 on dist-test-slave-lgr5"
server_key: "ca2ae5280cb5671eca5c069d031de1fe"
server_key_iv: "7806e4471c6e235cd30629de16d479eb"
server_key_version: "encryptionkey@0"
I20260806 12:24:07.794490 10454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:07.810093 10454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:07.813884 10454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:07.814136 10454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:07.814932 10454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:07.816244 10454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:07.816314 10454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:07.816367 10454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:07.816382 10454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:07.827663 10454 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:42421
I20260806 12:24:07.827685 10635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:42421 every 8 connection(s)
I20260806 12:24:07.829070 10454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:24:07.839075 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 10454
I20260806 12:24:07.839251 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:24:07.839598 29513 external_mini_cluster.cc:1428] Setting key e000cf02269f4d34e0762cb72937cbd4
Aug 06 12:24:07 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:24:07.845927 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:07.831820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:35887 (local address 127.28.210.126:34519)
0806 12:24:07.832061 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:07.832066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:07.833296 (+ 1230us) server_negotiation.cc:407] Connection header received
0806 12:24:07.834879 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:07.834884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:07.834961 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:07.835056 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:07.836591 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:07.837233 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.838571 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:07.838824 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.838921 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:07.842123 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:07.842127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:07.842130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:07.842163 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:07.843556 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:07.844163 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:07.844167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:07.844168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:07.844259 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:07.844638 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:07.844641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:07.844642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:07.844859 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:07.844954 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:07.845573 (+ 619us) server_negotiation.cc:299] Negotiation successful
0806 12:24:07.845743 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260806 12:24:07.846858 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:07.832397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:35887)
0806 12:24:07.833072 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:07.833129 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:07.834566 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:07.835305 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:07.835315 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:07.835764 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:07.836399 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:07.836418 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:07.837382 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.837385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:07.838439 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:07.838451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:07.839005 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:07.839010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:07.839205 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:07.839905 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:07.839928 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:07.841883 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:07.843689 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:07.843696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:07.843718 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:07.844026 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:07.844350 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:07.844354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:07.844356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:07.844529 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:07.844974 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:07.844983 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:07.845292 (+ 309us) client_negotiation.cc:769] Sending connection context
0806 12:24:07.845483 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 12:24:07.845884 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":181,"threads_started":1}
I20260806 12:24:07.848299 10636 heartbeater.cc:347] Connected to a master server at 127.28.210.126:34519
I20260806 12:24:07.848616 10636 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:07.849210 10636 heartbeater.cc:511] Master 127.28.210.126:34519 requested a full tablet report, sending...
I20260806 12:24:07.850488 9984 ts_manager.cc:194] Registered new tserver with Master: 45416e5017e041f09b50429dcac563ee (127.28.210.67:42421)
I20260806 12:24:07.851128 9984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:35887
I20260806 12:24:07.854148 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab.
Entry for principal oryx/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab.
WARNING: no policy specified for oryx/127.28.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab.
Entry for principal oryx/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab.
WARNING: no policy specified for oryx/127.28.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab.
Entry for principal oryx/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab.
I20260806 12:24:07.948550 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 10079
I20260806 12:24:07.983174 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 10270
I20260806 12:24:07.997781 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 10454
I20260806 12:24:08.010843 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 9944
I20260806 12:24:08.026117 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34519
--webserver_interface=127.28.210.126
--webserver_port=46811
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:34519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:08.155993 10650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:08.156371 10650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:08.156416 10650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:08.160792 10650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:24:08.160892 10650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:08.160912 10650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:08.160928 10650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:24:08.160941 10650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:24:08.166801 10650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:34519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=46811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:08.168188 10650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:08.169351 10650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:08.178923 10656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:08.178978 10650 server_base.cc:1034] running on GCE node
W20260806 12:24:08.178906 10657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:08.178936 10660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:08.179790 10650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:08.181108 10650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:08.182296 10650 hybrid_clock.cc:648] HybridClock initialized: now 1786019048182267 us; error 42 us; skew 500 ppm
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019048, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:08.185325 10650 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:24:08.186368 10650 webserver.cc:460] Webserver started at http://127.28.210.126:46811/ using document root <none> and password file <none>
I20260806 12:24:08.187002 10650 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:08.187100 10650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:08.191540 10650 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:24:08.194365 10674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:08.195912 10650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:24:08.196144 10650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "9024cf00daa6439198061ff0bdadc91f"
format_stamp: "Formatted at 2026-08-06 12:24:06 on dist-test-slave-lgr5"
server_key: "28f4a212cc5ee3f9023938a15be4fb04"
server_key_iv: "a7805e342815b3a90d0fac9ecf862fd2"
server_key_version: "encryptionkey@0"
I20260806 12:24:08.196833 10650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:08.216486 10650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:08.220427 10650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:08.220659 10650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:08.229354 10650 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:34519
I20260806 12:24:08.229516 10738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:34519 every 8 connection(s)
I20260806 12:24:08.230705 10650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:24:08.234894 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 10650
I20260806 12:24:08.235625 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:41217
--local_ip_for_outbound_sockets=127.28.210.65
--tserver_master_addrs=127.28.210.126:34519
--webserver_port=46087
--webserver_interface=127.28.210.65
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.210.65
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:24:08.237357 10740 sys_catalog.cc:263] Verifying existing consensus state
I20260806 12:24:08.239677 10740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f: Bootstrap starting.
I20260806 12:24:08.259160 10740 log.cc:824] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f: Log is configured to *not* fsync() on all Append() calls
I20260806 12:24:08.267433 10740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 12:24:08.267911 10740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f: Bootstrap complete.
I20260806 12:24:08.273232 10740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024cf00daa6439198061ff0bdadc91f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34519 } }
I20260806 12:24:08.273809 10740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9024cf00daa6439198061ff0bdadc91f, State: Initialized, Role: FOLLOWER
I20260806 12:24:08.274459 10740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024cf00daa6439198061ff0bdadc91f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34519 } }
I20260806 12:24:08.274618 10740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:24:08.274691 10740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:24:08.274801 10740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 1 FOLLOWER]: Advancing to term 2
I20260806 12:24:08.277307 10740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024cf00daa6439198061ff0bdadc91f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34519 } }
I20260806 12:24:08.277743 10740 leader_election.cc:302] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9024cf00daa6439198061ff0bdadc91f; no voters:
I20260806 12:24:08.278137 10740 leader_election.cc:288] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 12:24:08.278280 10749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 2 FOLLOWER]: Leader election won for term 2
I20260806 12:24:08.278504 10749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [term 2 LEADER]: Becoming Leader. State: Replica: 9024cf00daa6439198061ff0bdadc91f, State: Running, Role: LEADER
I20260806 12:24:08.278860 10749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024cf00daa6439198061ff0bdadc91f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34519 } }
I20260806 12:24:08.279675 10740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:24:08.280647 10750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9024cf00daa6439198061ff0bdadc91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024cf00daa6439198061ff0bdadc91f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34519 } } }
I20260806 12:24:08.280680 10751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9024cf00daa6439198061ff0bdadc91f. Latest consensus state: current_term: 2 leader_uuid: "9024cf00daa6439198061ff0bdadc91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9024cf00daa6439198061ff0bdadc91f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34519 } } }
I20260806 12:24:08.280804 10750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [sys.catalog]: This master's current role is: LEADER
I20260806 12:24:08.280807 10751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f [sys.catalog]: This master's current role is: LEADER
I20260806 12:24:08.281256 10765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:24:08.285285 10765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:24:08.286063 10765 catalog_manager.cc:1259] Loaded cluster ID: f27b4155ff9b42c1a40e322fe6a3b29d
I20260806 12:24:08.286141 10765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:24:08.288807 10765 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:24:08.289708 10765 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9024cf00daa6439198061ff0bdadc91f: Loaded TSK: 0
I20260806 12:24:08.291189 10765 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 12:24:08.370705 10746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:08.371037 10746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:08.371106 10746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:08.375775 10746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:08.375857 10746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:08.375947 10746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:24:08.381961 10746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.65.keytab
--principal=oryx/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:41217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=46087
--tserver_master_addrs=127.28.210.126:34519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:08.383173 10746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:08.384348 10746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:08.394757 10782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:08.394814 10781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:08.394821 10786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:08.395375 10746 server_base.cc:1034] running on GCE node
I20260806 12:24:08.395848 10746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:08.396669 10746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:08.397866 10746 hybrid_clock.cc:648] HybridClock initialized: now 1786019048397850 us; error 33 us; skew 500 ppm
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019048, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:08.401183 10746 init.cc:377] Logged in from keytab as oryx/127.28.210.65@KRBTEST.COM (short username oryx)
I20260806 12:24:08.402423 10746 webserver.cc:460] Webserver started at http://127.28.210.65:46087/ using document root <none> and password file <none>
I20260806 12:24:08.403067 10746 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:08.403170 10746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:08.407837 10746 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 12:24:08.410408 10794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:08.412034 10746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:24:08.412220 10746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "771bbb067a914c89b8ddfe9ee51ccf5f"
format_stamp: "Formatted at 2026-08-06 12:24:07 on dist-test-slave-lgr5"
server_key: "2d7ac9f20118012ce3021e1564d1f575"
server_key_iv: "1644db7e2c350fd55deb69f014f35cd4"
server_key_version: "encryptionkey@0"
I20260806 12:24:08.412837 10746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:08.441893 10746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:08.445423 10746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:08.445634 10746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:08.446305 10746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:08.447315 10746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:08.447371 10746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:08.447429 10746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:08.447459 10746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:08.456880 10746 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:41217
I20260806 12:24:08.456923 10982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:41217 every 8 connection(s)
I20260806 12:24:08.458117 10746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:24:08.465253 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 10746
I20260806 12:24:08.465749 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:42513
--local_ip_for_outbound_sockets=127.28.210.66
--tserver_master_addrs=127.28.210.126:34519
--webserver_port=38155
--webserver_interface=127.28.210.66
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.210.66
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:08.473577 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:08.460921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:42717)
0806 12:24:08.461456 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:08.461510 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:08.462773 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:08.464043 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:08.464052 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:08.464534 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:08.465047 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.465063 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.466568 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.466571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.467588 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.467600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.468588 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.468592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.468800 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.469484 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:08.469504 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:08.472311 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":132,"threads_started":1}
W20260806 12:24:08.473567 10987 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:08.460774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:42717 (local address 127.28.210.126:34519)
0806 12:24:08.461403 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:08.461423 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:08.461632 (+ 209us) server_negotiation.cc:407] Connection header received
0806 12:24:08.463220 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:08.463231 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:08.463571 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:08.463893 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:08.465234 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.466403 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.467740 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.468368 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.468577 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.472661 (+ 4084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:42717: BlockingRecv error: recv got EOF from 127.28.210.65:42717 (error 108)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":190,"threads_started":1}
W20260806 12:24:08.474457 10983 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:34519 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.210.126:34519: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:08.480470 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:08.475094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:36765)
0806 12:24:08.475248 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:08.475264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:08.475380 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:08.475692 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:08.475695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:08.475875 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:08.475993 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.475999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.476827 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.476830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.477394 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.477403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.477855 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.477858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.477950 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.478445 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:08.478457 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:08.480311 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 12:24:08.480633 10987 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:08.475163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:36765 (local address 127.28.210.126:34519)
0806 12:24:08.475281 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:08.475285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:08.475318 (+ 33us) server_negotiation.cc:407] Connection header received
0806 12:24:08.475517 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:08.475521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:08.475577 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:08.475654 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:08.476145 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.476715 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.477517 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.477744 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.477834 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.480518 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:36765: BlockingRecv error: recv got EOF from 127.28.210.65:36765 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:08.486693 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:08.481277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:57801)
0806 12:24:08.481417 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:08.481432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:08.481572 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:08.481814 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:08.481817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:08.481968 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:08.482052 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.482058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.483046 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.483049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.483594 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.483604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.484045 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.484049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.484160 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.484643 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:08.484656 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:08.486574 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 12:24:08.486922 10983 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 12:24:08.487262 10987 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:08.481346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:57801 (local address 127.28.210.126:34519)
0806 12:24:08.481467 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:08.481471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:08.481484 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:24:08.481667 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:08.481670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:08.481716 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:08.481781 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:08.482271 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.482917 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.483713 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.483935 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.484025 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.486763 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:57801: BlockingRecv error: recv got EOF from 127.28.210.65:57801 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 12:24:08.591355 10989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:08.591641 10989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:08.591687 10989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:08.595984 10989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:08.596066 10989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:08.596166 10989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:24:08.601877 10989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.66.keytab
--principal=oryx/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:42513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=38155
--tserver_master_addrs=127.28.210.126:34519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:08.603097 10989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:08.604209 10989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:08.614807 10994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:08.614811 10995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:08.614811 10997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:08.615264 10989 server_base.cc:1034] running on GCE node
I20260806 12:24:08.615763 10989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:08.616639 10989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:08.617811 10989 hybrid_clock.cc:648] HybridClock initialized: now 1786019048617789 us; error 36 us; skew 500 ppm
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019048, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:08.621063 10989 init.cc:377] Logged in from keytab as oryx/127.28.210.66@KRBTEST.COM (short username oryx)
I20260806 12:24:08.622575 10989 webserver.cc:460] Webserver started at http://127.28.210.66:38155/ using document root <none> and password file <none>
I20260806 12:24:08.623435 10989 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:08.623565 10989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:08.628679 10989 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:24:08.631261 11009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:08.632812 10989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:24:08.632964 10989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "dbeb51f14fec4829b7520ba8bcb27cd9"
format_stamp: "Formatted at 2026-08-06 12:24:07 on dist-test-slave-lgr5"
server_key: "4e630ec3cf2221e71ebe659a8652e98f"
server_key_iv: "85da24d4d798f3cd6798c644e5748f96"
server_key_version: "encryptionkey@0"
I20260806 12:24:08.633622 10989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:08.651490 10989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:08.655406 10989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:08.655635 10989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:08.656373 10989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:08.657353 10989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:08.657410 10989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:08.657473 10989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:08.657505 10989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:08.667342 10989 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:42513
I20260806 12:24:08.667356 11169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:42513 every 8 connection(s)
I20260806 12:24:08.668577 10989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:24:08.675549 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 10989
I20260806 12:24:08.676036 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:42421
--local_ip_for_outbound_sockets=127.28.210.67
--tserver_master_addrs=127.28.210.126:34519
--webserver_port=38901
--webserver_interface=127.28.210.67
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.210.67
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 12:24:08.681267 10987 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:08.670638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:35143 (local address 127.28.210.126:34519)
0806 12:24:08.670773 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:08.670777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:08.671719 (+ 942us) server_negotiation.cc:407] Connection header received
0806 12:24:08.673103 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:08.673106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:08.673166 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:08.673247 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:08.674532 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.675127 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.676543 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.676876 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.677000 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.681124 (+ 4124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:35143: BlockingRecv error: recv got EOF from 127.28.210.66:35143 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 12:24:08.682065 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:08.671035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:35143)
0806 12:24:08.671529 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:08.671577 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:08.672856 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:08.673421 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:08.673428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:08.673819 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:08.674354 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.674369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.675284 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.675287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.676391 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.676404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.677083 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.677086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.677260 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.677925 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:08.677952 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:08.680975 (+ 3023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
W20260806 12:24:08.682778 11171 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:34519 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.210.126:34519: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:08.689728 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:08.683321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:53941)
0806 12:24:08.683463 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:08.683480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:08.683581 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:08.683889 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:08.683909 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:08.684106 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:08.684225 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.684232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.685281 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.685284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.685876 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.685888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.686455 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.686458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.686537 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.687164 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:08.687178 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:08.689574 (+ 2396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 12:24:08.689960 10987 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:08.683456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:53941 (local address 127.28.210.126:34519)
0806 12:24:08.683597 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:08.683602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:08.683623 (+ 21us) server_negotiation.cc:407] Connection header received
0806 12:24:08.683682 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:08.683685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:08.683741 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:08.683835 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:08.684350 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.685149 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.686002 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.686341 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.686456 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.689808 (+ 3352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:53941: BlockingRecv error: recv got EOF from 127.28.210.66:53941 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:08.697331 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:08.690434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:36933)
0806 12:24:08.690557 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:08.690573 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:08.690670 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:08.690940 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:08.690942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:08.691103 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:08.691203 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.691209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.692533 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.692538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.693188 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.693204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.693997 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.694002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.694090 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.694793 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:08.694808 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:08.697146 (+ 2338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 12:24:08.697644 11171 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 12:24:08.697701 10987 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:08.690522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:36933 (local address 127.28.210.126:34519)
0806 12:24:08.690638 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:08.690643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:08.690660 (+ 17us) server_negotiation.cc:407] Connection header received
0806 12:24:08.690755 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:08.690758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:08.690811 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:08.690901 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:08.691310 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.692338 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.693334 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.693785 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.693925 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.697341 (+ 3416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:36933: BlockingRecv error: recv got EOF from 127.28.210.66:36933 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 12:24:08.809844 11174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:08.810194 11174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:08.810243 11174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:08.815009 11174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:08.815124 11174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:08.815191 11174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:24:08.821560 11174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/krb5kdc/oryx_127.28.210.67.keytab
--principal=oryx/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:42421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=38901
--tserver_master_addrs=127.28.210.126:34519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:08.822940 11174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:08.824155 11174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:08.835630 11185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:08.835637 11181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:08.835637 11182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:08.835877 11174 server_base.cc:1034] running on GCE node
I20260806 12:24:08.836388 11174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:08.837109 11174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:08.838305 11174 hybrid_clock.cc:648] HybridClock initialized: now 1786019048838285 us; error 34 us; skew 500 ppm
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019048, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:08.841441 11174 init.cc:377] Logged in from keytab as oryx/127.28.210.67@KRBTEST.COM (short username oryx)
I20260806 12:24:08.842451 11174 webserver.cc:460] Webserver started at http://127.28.210.67:38901/ using document root <none> and password file <none>
I20260806 12:24:08.843050 11174 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:08.843127 11174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:08.847496 11174 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:24:08.849999 11194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:08.851488 11174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:24:08.851620 11174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "45416e5017e041f09b50429dcac563ee"
format_stamp: "Formatted at 2026-08-06 12:24:07 on dist-test-slave-lgr5"
server_key: "ca2ae5280cb5671eca5c069d031de1fe"
server_key_iv: "7806e4471c6e235cd30629de16d479eb"
server_key_version: "encryptionkey@0"
I20260806 12:24:08.852313 11174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:08.869933 11174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:08.873364 11174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:08.873561 11174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:08.874184 11174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:08.875080 11174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:08.875129 11174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:08.875172 11174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:08.875216 11174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:08.885792 11174 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:42421
I20260806 12:24:08.885819 11357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:42421 every 8 connection(s)
I20260806 12:24:08.887116 11174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:24:08.895881 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 11174
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 12:24:08.900483 10987 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:08.889515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:51139 (local address 127.28.210.126:34519)
0806 12:24:08.889721 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:08.889725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:08.890701 (+ 976us) server_negotiation.cc:407] Connection header received
0806 12:24:08.892367 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:08.892371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:08.892437 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:08.892522 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:08.893931 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.894488 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.895758 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.895987 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.896108 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.900344 (+ 4236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:51139: BlockingRecv error: recv got EOF from 127.28.210.67:51139 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 12:24:08.901402 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:08.889928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:51139)
0806 12:24:08.890504 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:08.890562 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:08.892050 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:08.892739 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:08.892748 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:08.893190 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:08.893752 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.893767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.894627 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.894630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.895619 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.895631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.896180 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.896185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.896376 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.897123 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:08.897152 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:08.900207 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":133,"threads_started":1}
W20260806 12:24:08.902122 11358 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:34519 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.210.126:34519: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:08.907263 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:08.897325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.28.210.126:34519)
0806 12:24:08.897769 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:08.897772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:08.897786 (+ 14us) server_negotiation.cc:407] Connection header received
0806 12:24:08.897826 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:08.897828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:08.897868 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:08.897966 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:08.898839 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.899573 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.900526 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.900906 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.901015 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.902568 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:08.902573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:08.902590 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:08.902626 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:08.904866 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:08.905445 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:08.905452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:08.905459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:08.905609 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:08.905896 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:08.905900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:08.905901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:08.906306 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:08.906527 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:08.906815 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 12:24:08.907123 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":120,"threads_started":1}
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:08.908068 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:08.902664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:43999)
0806 12:24:08.902790 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:08.902805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:08.902887 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:08.903117 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:08.903121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:08.903295 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:08.903406 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.903412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.904478 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.904481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.905076 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.905088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.905553 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.905557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.905658 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.906089 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:08.906102 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:08.907945 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 12:24:08.908246 10987 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:08.902753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:43999 (local address 127.28.210.126:34519)
0806 12:24:08.902888 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:08.902892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:08.902906 (+ 14us) server_negotiation.cc:407] Connection header received
0806 12:24:08.902963 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:08.902965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:08.903009 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:08.903076 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:08.903563 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.904328 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.905213 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.905439 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.905533 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.908130 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:43999: BlockingRecv error: recv got EOF from 127.28.210.67:43999 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:08 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:08.913678 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:08.908930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:39417)
0806 12:24:08.909032 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:08.909046 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:08.909126 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:08.909332 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:08.909335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:08.909484 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:08.909560 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.909565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.910341 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.910344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.910873 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:08.910881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.911310 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.911314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:08.911390 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.911822 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:08.911834 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:08.913571 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 12:24:08.913841 10987 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:08.909013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:39417 (local address 127.28.210.126:34519)
0806 12:24:08.909150 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:08.909154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:08.909167 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:24:08.909200 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:08.909203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:08.909243 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:08.909303 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:08.909666 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.910236 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.910982 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:08.911215 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:08.911298 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:08.913714 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:39417: BlockingRecv error: recv got EOF from 127.28.210.67:39417 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 12:24:08.913930 11358 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 12:24:09 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:09 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:09.493204 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:09.487564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:56495)
0806 12:24:09.487860 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:09.487875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:09.487973 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:09.488220 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:09.488222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:09.488411 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:09.488514 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:09.488519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:09.489353 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:09.489376 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:09.489970 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:09.489981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:09.490477 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:09.490480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:09.490549 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:09.491072 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:09.491086 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:09.493061 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
W20260806 12:24:09.493407 11365 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:09.487638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:56495 (local address 127.28.210.126:34519)
0806 12:24:09.487903 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:09.487906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:09.487922 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:24:09.488050 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:09.488052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:09.488119 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:09.488194 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:09.488668 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:09.489217 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:09.490110 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:09.490353 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:09.490443 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:09.493247 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:56495: BlockingRecv error: recv got EOF from 127.28.210.65:56495 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
Aug 06 12:24:09 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:09 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:09.704730 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:09.698436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:59099)
0806 12:24:09.698736 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:09.698750 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:09.698862 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:09.699131 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:09.699134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:09.699351 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:09.699457 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:09.699463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:09.700360 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:09.700363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:09.700983 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:09.700995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:09.701518 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:09.701521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:09.701591 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:09.702174 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:09.702187 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:09.704551 (+ 2364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
W20260806 12:24:09.704823 11365 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:09.698593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:59099 (local address 127.28.210.126:34519)
0806 12:24:09.698742 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:09.698746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:09.698817 (+ 71us) server_negotiation.cc:407] Connection header received
0806 12:24:09.698946 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:09.698948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:09.698997 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:09.699081 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:09.699595 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:09.700215 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:09.701129 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:09.701374 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:09.701470 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:09.704716 (+ 3246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:59099: BlockingRecv error: recv got EOF from 127.28.210.66:59099 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 12:24:09 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:09 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:09.920995 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:09.914692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:48685)
0806 12:24:09.915011 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:09.915040 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:09.915167 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:09.915430 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:09.915433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:09.915648 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:09.915748 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:09.915754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:09.916582 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:09.916586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:09.917256 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:09.917267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:09.917733 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:09.917736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:09.917818 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:09.918428 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:09.918441 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:09.920754 (+ 2313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
W20260806 12:24:09.921154 11365 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:09.914782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:48685 (local address 127.28.210.126:34519)
0806 12:24:09.914914 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:09.914919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:09.915121 (+ 202us) server_negotiation.cc:407] Connection header received
0806 12:24:09.915259 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:09.915262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:09.915308 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:09.915371 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:09.915870 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:09.916460 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:09.917377 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:09.917614 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:09.917709 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:09.921020 (+ 3311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:48685: BlockingRecv error: recv got EOF from 127.28.210.67:48685 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 12:24:10 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:10 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:10.503756 11370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:10.494417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:58313)
0806 12:24:10.494792 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:10.494809 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:10.494940 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:10.495287 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:10.495290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:10.495513 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:10.495625 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:10.495632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:10.496533 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:10.496536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:10.497114 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:10.497123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:10.497649 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:10.497651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:10.497714 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:10.501537 (+ 3823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:10.501554 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:10.503539 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
W20260806 12:24:10.503859 11371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:10.494518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:58313 (local address 127.28.210.126:34519)
0806 12:24:10.494790 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:10.494794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:10.494961 (+ 167us) server_negotiation.cc:407] Connection header received
0806 12:24:10.495035 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:10.495037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:10.495091 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:10.495189 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:10.495763 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:10.496376 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:10.497253 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:10.497541 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:10.497635 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:10.503731 (+ 6096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:58313: BlockingRecv error: recv got EOF from 127.28.210.65:58313 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
Aug 06 12:24:10 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:10 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:10.712412 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:10.706013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:53277)
0806 12:24:10.706398 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:10.706417 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:10.706558 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:10.706910 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:10.706913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:10.707230 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:10.707358 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:10.707366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:10.708415 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:10.708418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:10.709044 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:10.709055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:10.709513 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:10.709516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:10.709579 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:10.710126 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:10.710149 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:10.712248 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":134,"threads_started":1}
W20260806 12:24:10.712628 11371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:10.706174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:53277 (local address 127.28.210.126:34519)
0806 12:24:10.706344 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:10.706349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:10.706554 (+ 205us) server_negotiation.cc:407] Connection header received
0806 12:24:10.706649 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:10.706653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:10.706714 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:10.706808 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:10.707518 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:10.708288 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:10.709170 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:10.709413 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:10.709508 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:10.712482 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:53277: BlockingRecv error: recv got EOF from 127.28.210.66:53277 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 12:24:10 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:10 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:10.928231 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:10.921874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:55619)
0806 12:24:10.922177 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:10.922191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:10.922299 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:10.922530 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:10.922533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:10.922733 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:10.922845 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:10.922851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:10.923737 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:10.923740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:10.924408 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:10.924423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:10.925035 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:10.925038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:10.925112 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:10.925694 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:10.925709 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:10.928007 (+ 2298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":106,"threads_started":1}
W20260806 12:24:10.928623 11371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:10.922052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:55619 (local address 127.28.210.126:34519)
0806 12:24:10.922190 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:10.922194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:10.922255 (+ 61us) server_negotiation.cc:407] Connection header received
0806 12:24:10.922375 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:10.922377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:10.922427 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:10.922518 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:10.922944 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:10.923514 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:10.924603 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:10.924867 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:10.924966 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:10.928459 (+ 3493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:55619: BlockingRecv error: recv got EOF from 127.28.210.67:55619 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 12:24:11 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:11 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:11.511117 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:11.504968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:59961)
0806 12:24:11.505440 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:11.505463 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:11.505641 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:11.505959 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:11.505962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:11.506209 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:11.506347 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:11.506356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:11.507198 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:11.507202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:11.507789 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:11.507799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:11.508389 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:11.508392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:11.508478 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:11.509029 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:11.509042 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:11.510983 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":92,"threads_started":1}
W20260806 12:24:11.511386 11375 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:11.505056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:59961 (local address 127.28.210.126:34519)
0806 12:24:11.505419 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:11.505426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:11.505601 (+ 175us) server_negotiation.cc:407] Connection header received
0806 12:24:11.505748 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:11.505751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:11.505816 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:11.505919 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:11.506480 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:11.507059 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:11.507944 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:11.508233 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:11.508346 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:11.511245 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:59961: BlockingRecv error: recv got EOF from 127.28.210.65:59961 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":121,"threads_started":1}
Aug 06 12:24:11 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:11 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:11.719101 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:11.713288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:43803)
0806 12:24:11.713558 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:11.713572 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:11.713690 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:11.713993 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:11.713996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:11.714206 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:11.714326 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:11.714332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:11.715168 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:11.715171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:11.715756 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:11.715765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:11.716314 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:11.716318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:11.716412 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:11.716982 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:11.716994 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:11.718966 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
W20260806 12:24:11.719358 11375 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:11.713384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:43803 (local address 127.28.210.126:34519)
0806 12:24:11.713574 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:11.713578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:11.713637 (+ 59us) server_negotiation.cc:407] Connection header received
0806 12:24:11.713813 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:11.713816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:11.713873 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:11.713949 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:11.714457 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:11.715042 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:11.715895 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:11.716171 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:11.716287 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:11.719233 (+ 2946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:43803: BlockingRecv error: recv got EOF from 127.28.210.66:43803 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 12:24:11 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:11 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:11.936645 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:11.929396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:52289)
0806 12:24:11.929805 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:11.929823 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:11.929971 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:11.930250 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:11.930253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:11.930484 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:11.930590 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:11.930596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:11.931404 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:11.931407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:11.931989 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:11.931998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:11.932512 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:11.932515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:11.932599 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:11.933161 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:11.933174 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:11.936500 (+ 3326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":136,"threads_started":1}
W20260806 12:24:11.936822 11375 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:11.929574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:52289 (local address 127.28.210.126:34519)
0806 12:24:11.929790 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:11.929796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:11.929912 (+ 116us) server_negotiation.cc:407] Connection header received
0806 12:24:11.930074 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:11.930077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:11.930135 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:11.930209 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:11.930721 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:11.931287 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:11.932149 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:11.932393 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:11.932491 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:11.936687 (+ 4196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:52289: BlockingRecv error: recv got EOF from 127.28.210.67:52289 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 06 12:24:12 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:12 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:12.519222 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:12.512333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:53387)
0806 12:24:12.512715 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:12.512731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:12.512858 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:12.513121 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:12.513123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:12.513348 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:12.513458 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:12.513464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:12.514270 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:12.514273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:12.514972 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:12.514990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:12.515865 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:12.515869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:12.515967 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:12.516697 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:12.516715 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:12.519050 (+ 2335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":93,"threads_started":1}
W20260806 12:24:12.519474 11379 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:12.512495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:53387 (local address 127.28.210.126:34519)
0806 12:24:12.512800 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:12.512804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:12.512821 (+ 17us) server_negotiation.cc:407] Connection header received
0806 12:24:12.512947 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:12.512949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:12.513000 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:12.513087 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:12.513560 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:12.514128 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:12.515283 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:12.515712 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:12.515847 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:12.519307 (+ 3460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:53387: BlockingRecv error: recv got EOF from 127.28.210.65:53387 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":98,"threads_started":1}
Aug 06 12:24:12 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:12 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:12.726352 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:12.720041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:45225)
0806 12:24:12.720378 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:12.720396 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:12.720570 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:12.720845 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:12.720849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:12.721082 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:12.721188 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:12.721194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:12.721990 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:12.721994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:12.722595 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:12.722606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:12.723095 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:12.723099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:12.723182 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:12.723873 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:12.723889 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:12.726170 (+ 2281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":96,"threads_started":1}
W20260806 12:24:12.726470 11379 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:12.720202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:45225 (local address 127.28.210.126:34519)
0806 12:24:12.720381 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:12.720386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:12.720507 (+ 121us) server_negotiation.cc:407] Connection header received
0806 12:24:12.720673 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:12.720676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:12.720728 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:12.720800 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:12.721328 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:12.721872 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:12.722730 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:12.722972 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:12.723070 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:12.726358 (+ 3288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:45225: BlockingRecv error: recv got EOF from 127.28.210.66:45225 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 12:24:12 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:12 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:12.944412 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:12.937661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:53877)
0806 12:24:12.937977 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:12.938006 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:12.938219 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:12.938650 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:12.938653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:12.938930 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:12.939052 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:12.939059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:12.939949 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:12.939952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:12.940679 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:12.940695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:12.941212 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:12.941215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:12.941287 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:12.941867 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:12.941880 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:12.944238 (+ 2358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
W20260806 12:24:12.944660 11379 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:12.937778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:53877 (local address 127.28.210.126:34519)
0806 12:24:12.937946 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:12.937952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:12.938238 (+ 286us) server_negotiation.cc:407] Connection header received
0806 12:24:12.938343 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:12.938347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:12.938443 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:12.938535 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:12.939215 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:12.939816 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:12.940821 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:12.941086 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:12.941188 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:12.944509 (+ 3321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:53877: BlockingRecv error: recv got EOF from 127.28.210.67:53877 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 12:24:13 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:13 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:13.526446 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:13.520419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:47049)
0806 12:24:13.520734 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:13.520751 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:13.520866 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:13.521131 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:13.521134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:13.521370 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:13.521510 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:13.521519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:13.522430 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:13.522434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:13.523176 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:13.523188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:13.523702 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:13.523705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:13.523782 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:13.524395 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:13.524408 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:13.526277 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":85,"threads_started":1}
W20260806 12:24:13.526679 11383 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:13.520641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:47049 (local address 127.28.210.126:34519)
0806 12:24:13.520889 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:13.520893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:13.520924 (+ 31us) server_negotiation.cc:407] Connection header received
0806 12:24:13.520985 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:13.520988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:13.521031 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:13.521100 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:13.521681 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:13.522308 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:13.523291 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:13.523550 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:13.523648 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:13.526443 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:47049: BlockingRecv error: recv got EOF from 127.28.210.65:47049 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
Aug 06 12:24:13 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:13 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:13.735405 11384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:13.727714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:49769)
0806 12:24:13.728147 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:13.728163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:13.728274 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:13.728646 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:13.728649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:13.728853 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:13.728967 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:13.728972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:13.730161 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:13.730167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:13.730899 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:13.730918 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:13.731670 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:13.731673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:13.731784 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:13.732748 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:13.732769 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:13.735232 (+ 2463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":184,"threads_started":1}
W20260806 12:24:13.735528 11383 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:13.728022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:49769 (local address 127.28.210.126:34519)
0806 12:24:13.728197 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:13.728200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:13.728218 (+ 18us) server_negotiation.cc:407] Connection header received
0806 12:24:13.728406 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:13.728409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:13.728460 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:13.728546 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:13.729232 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:13.729908 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:13.731122 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:13.731462 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:13.731619 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:13.735411 (+ 3792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:49769: BlockingRecv error: recv got EOF from 127.28.210.66:49769 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 12:24:13 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:13 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:13.951226 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:13.945418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:52093)
0806 12:24:13.945730 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:13.945744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:13.945852 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:13.946179 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:13.946181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:13.946403 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:13.946517 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:13.946524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:13.947393 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:13.947397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:13.947978 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:13.947988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:13.948494 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:13.948498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:13.948569 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:13.949160 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:13.949173 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:13.951084 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":105,"threads_started":1}
W20260806 12:24:13.951442 11383 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:13.945547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:52093 (local address 127.28.210.126:34519)
0806 12:24:13.945748 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:13.945753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:13.945816 (+ 63us) server_negotiation.cc:407] Connection header received
0806 12:24:13.945930 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:13.945932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:13.945983 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:13.946061 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:13.946677 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:13.947256 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:13.948139 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:13.948370 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:13.948458 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:13.951291 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:52093: BlockingRecv error: recv got EOF from 127.28.210.67:52093 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 06 12:24:14 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:14 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:14.533819 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:14.527558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:54255)
0806 12:24:14.527883 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:14.527898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:14.528022 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:14.528292 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:14.528294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:14.528526 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:14.528653 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:14.528662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:14.529630 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:14.529634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:14.530333 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:14.530356 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:14.531096 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:14.531099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:14.531175 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:14.531732 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:14.531745 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:14.533671 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":103,"threads_started":1}
W20260806 12:24:14.534355 11387 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:14.527721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:54255 (local address 127.28.210.126:34519)
0806 12:24:14.528014 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:14.528018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:14.528036 (+ 18us) server_negotiation.cc:407] Connection header received
0806 12:24:14.528123 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:14.528125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:14.528172 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:14.528258 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:14.528773 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:14.529451 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:14.530606 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:14.530959 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:14.531049 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:14.534228 (+ 3179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:54255: BlockingRecv error: recv got EOF from 127.28.210.65:54255 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":106,"threads_started":1}
Aug 06 12:24:14 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:14 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:14.742805 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:14.736474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:52845)
0806 12:24:14.736806 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:14.736821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:14.736945 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:14.737207 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:14.737209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:14.737432 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:14.737563 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:14.737571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:14.738432 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:14.738435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:14.739100 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:14.739113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:14.739615 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:14.739618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:14.739686 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:14.740310 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:14.740323 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:14.742670 (+ 2347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
W20260806 12:24:14.743000 11387 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:14.736643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:52845 (local address 127.28.210.126:34519)
0806 12:24:14.736830 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:14.736834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:14.736910 (+ 76us) server_negotiation.cc:407] Connection header received
0806 12:24:14.737047 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:14.737050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:14.737102 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:14.737177 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:14.737690 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:14.738289 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:14.739234 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:14.739507 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:14.739597 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:14.742859 (+ 3262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:52845: BlockingRecv error: recv got EOF from 127.28.210.66:52845 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 12:24:14 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:14 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:14.960850 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:14.952308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:45261)
0806 12:24:14.952714 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:14.952728 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:14.952863 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:14.953367 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:14.953371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:14.953615 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:14.953751 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:14.953759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:14.954693 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:14.954699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:14.955568 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:14.955623 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:14.956441 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:14.956446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:14.956561 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:14.957344 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:14.957362 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:14.960678 (+ 3316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":126,"threads_started":1}
W20260806 12:24:14.961141 11387 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:14.952472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:45261 (local address 127.28.210.126:34519)
0806 12:24:14.952708 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:14.952714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:14.952816 (+ 102us) server_negotiation.cc:407] Connection header received
0806 12:24:14.953071 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:14.953076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:14.953151 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:14.953242 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:14.953918 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:14.954503 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:14.955931 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:14.956266 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:14.956374 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:14.960970 (+ 4596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:45261: BlockingRecv error: recv got EOF from 127.28.210.67:45261 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 06 12:24:15 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:15 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:15.544901 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:15.534922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:37191)
0806 12:24:15.535352 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:15.535367 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:15.535476 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:15.535731 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:15.535733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:15.535929 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:15.536045 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:15.536051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:15.536957 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:15.536960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:15.537616 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:15.537629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:15.538207 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:15.538211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:15.538305 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:15.542635 (+ 4330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:15.542654 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:15.544753 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":107,"threads_started":1}
W20260806 12:24:15.545152 11391 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:15.535109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:37191 (local address 127.28.210.126:34519)
0806 12:24:15.535419 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:15.535423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:15.535440 (+ 17us) server_negotiation.cc:407] Connection header received
0806 12:24:15.535567 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:15.535570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:15.535619 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:15.535699 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:15.536195 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:15.536806 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:15.537740 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:15.538073 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:15.538184 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:15.544994 (+ 6810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:37191: BlockingRecv error: recv got EOF from 127.28.210.65:37191 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":129,"threads_started":1}
Aug 06 12:24:15 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:15 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:15.749573 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:15.743711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:40899)
0806 12:24:15.744048 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:15.744061 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:15.744211 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:15.744470 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:15.744473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:15.744713 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:15.744809 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:15.744815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:15.745608 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:15.745612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:15.746197 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:15.746206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:15.746668 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:15.746672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:15.746753 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:15.747314 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:15.747327 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:15.749380 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":95,"threads_started":1}
W20260806 12:24:15.749688 11391 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:15.743790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:40899 (local address 127.28.210.126:34519)
0806 12:24:15.743944 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:15.743950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:15.744163 (+ 213us) server_negotiation.cc:407] Connection header received
0806 12:24:15.744304 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:15.744307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:15.744359 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:15.744422 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:15.744926 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:15.745490 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:15.746313 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:15.746550 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:15.746644 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:15.749578 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:40899: BlockingRecv error: recv got EOF from 127.28.210.66:40899 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 12:24:15 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:15 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:15.968276 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:15.961860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:58315)
0806 12:24:15.962197 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:15.962214 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:15.962346 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:15.962725 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:15.962728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:15.962956 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:15.963097 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:15.963106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:15.963998 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:15.964002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:15.964660 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:15.964670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:15.965151 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:15.965154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:15.965214 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:15.965771 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:15.965783 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:15.968125 (+ 2342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":90,"threads_started":1}
W20260806 12:24:15.968458 11391 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:15.961980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:58315 (local address 127.28.210.126:34519)
0806 12:24:15.962109 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:15.962114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:15.962359 (+ 245us) server_negotiation.cc:407] Connection header received
0806 12:24:15.962450 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:15.962453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:15.962524 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:15.962639 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:15.963240 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:15.963872 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:15.964798 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:15.965045 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:15.965135 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:15.968322 (+ 3187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:58315: BlockingRecv error: recv got EOF from 127.28.210.67:58315 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 06 12:24:16 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:16 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:16.551770 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:16.545807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:55117)
0806 12:24:16.546114 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:16.546128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:16.546224 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:16.546446 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:16.546448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:16.546664 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:16.546781 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:16.546787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:16.547611 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:16.547614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:16.548263 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:16.548275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:16.548892 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:16.548896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:16.548973 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:16.549565 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:16.549578 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:16.551626 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":88,"threads_started":1}
W20260806 12:24:16.551960 11395 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:16.545949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:55117 (local address 127.28.210.126:34519)
0806 12:24:16.546193 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:16.546197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:16.546211 (+ 14us) server_negotiation.cc:407] Connection header received
0806 12:24:16.546295 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:16.546297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:16.546342 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:16.546418 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:16.546901 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:16.547489 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:16.548390 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:16.548728 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:16.548822 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:16.551819 (+ 2997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:55117: BlockingRecv error: recv got EOF from 127.28.210.65:55117 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
Aug 06 12:24:16 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:16 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:16.757542 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:16.750791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:44213)
0806 12:24:16.751138 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:16.751155 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:16.751321 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:16.751646 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:16.751650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:16.751925 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:16.752037 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:16.752044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:16.753078 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:16.753081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:16.753690 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:16.753699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:16.754203 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:16.754206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:16.754286 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:16.754868 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:16.754883 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:16.757385 (+ 2502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
W20260806 12:24:16.757763 11395 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:16.750918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:44213 (local address 127.28.210.126:34519)
0806 12:24:16.751092 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:16.751098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:16.751239 (+ 141us) server_negotiation.cc:407] Connection header received
0806 12:24:16.751450 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:16.751453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:16.751507 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:16.751591 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:16.752230 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:16.752954 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:16.753800 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:16.754054 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:16.754136 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:16.757626 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:44213: BlockingRecv error: recv got EOF from 127.28.210.66:44213 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 12:24:16 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:16 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:16.975816 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:16.969192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:40751)
0806 12:24:16.969571 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:16.969589 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:16.969744 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:16.970029 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:16.970032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:16.970252 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:16.970404 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:16.970411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:16.971265 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:16.971269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:16.971883 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:16.971894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:16.972577 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:16.972581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:16.972669 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:16.973250 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:16.973263 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:16.975639 (+ 2376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":100,"threads_started":1}
W20260806 12:24:16.976172 11395 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:16.969378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:40751 (local address 127.28.210.126:34519)
0806 12:24:16.969573 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:16.969578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:16.969672 (+ 94us) server_negotiation.cc:407] Connection header received
0806 12:24:16.969856 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:16.969859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:16.969913 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:16.969987 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:16.970539 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:16.971111 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:16.972026 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:16.972406 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:16.972569 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:16.975962 (+ 3393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:40751: BlockingRecv error: recv got EOF from 127.28.210.67:40751 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 12:24:17 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:17 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:17.559420 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:17.552963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:57277)
0806 12:24:17.553431 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:17.553445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:17.553599 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:17.553970 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:17.553973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:17.554216 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:17.554342 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:17.554350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:17.555289 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:17.555293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:17.555918 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:17.555928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:17.556520 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:17.556523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:17.556673 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:17.557361 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:17.557374 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:17.559259 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":92,"threads_started":1}
W20260806 12:24:17.559670 11400 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:17.553098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:57277 (local address 127.28.210.126:34519)
0806 12:24:17.553441 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:17.553447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:17.553531 (+ 84us) server_negotiation.cc:407] Connection header received
0806 12:24:17.553757 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:17.553760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:17.553813 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:17.553925 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:17.554493 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:17.555132 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:17.556116 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:17.556371 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:17.556469 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:17.559521 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:57277: BlockingRecv error: recv got EOF from 127.28.210.65:57277 (error 108)
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":100,"threads_started":1}
Aug 06 12:24:17 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:17 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:17.764832 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:17.758716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:45229)
0806 12:24:17.759013 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:17.759027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:17.759145 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:17.759384 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:17.759386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:17.759576 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:17.759684 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:17.759690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:17.760566 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:17.760569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:17.761186 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:17.761198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:17.761705 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:17.761709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:17.761788 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:17.762372 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:17.762386 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:17.764639 (+ 2253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
W20260806 12:24:17.765202 11400 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:17.758840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:45229 (local address 127.28.210.126:34519)
0806 12:24:17.759010 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:17.759014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:17.759107 (+ 93us) server_negotiation.cc:407] Connection header received
0806 12:24:17.759226 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:17.759228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:17.759278 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:17.759360 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:17.759833 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:17.760417 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:17.761324 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:17.761569 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:17.761655 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:17.764892 (+ 3237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:45229: BlockingRecv error: recv got EOF from 127.28.210.66:45229 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 12:24:17 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:17 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:17.985827 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:17.979099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:58179)
0806 12:24:17.979483 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:17.979501 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:17.979616 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:17.979944 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:17.979947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:17.980196 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:17.980323 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:17.980331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:17.981349 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:17.981352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:17.982195 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:17.982208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:17.982761 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:17.982765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:17.982851 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:17.983410 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:17.983426 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:17.985683 (+ 2257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":91,"threads_started":1}
W20260806 12:24:17.985936 11400 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:17.979559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:58179 (local address 127.28.210.126:34519)
0806 12:24:17.979687 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:17.979692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:17.979707 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:24:17.979755 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:17.979758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:17.979815 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:17.979902 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:17.980488 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:17.981227 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:17.982310 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:17.982635 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:17.982727 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:17.985830 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:58179: BlockingRecv error: recv got EOF from 127.28.210.67:58179 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 12:24:18 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:18 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:18.566589 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:18.560584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:36815)
0806 12:24:18.560950 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:18.560966 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:18.561076 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:18.561330 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:18.561332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:18.561532 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:18.561641 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:18.561647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:18.562439 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:18.562443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:18.563048 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:18.563057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:18.563551 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:18.563554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:18.563629 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:18.564260 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:18.564275 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:18.566444 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":104,"threads_started":1}
W20260806 12:24:18.566766 11406 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:18.560809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:36815 (local address 127.28.210.126:34519)
0806 12:24:18.561054 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:18.561058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:18.561074 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:24:18.561166 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:18.561168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:18.561214 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:18.561286 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:18.561757 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:18.562341 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:18.563172 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:18.563410 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:18.563498 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:18.566621 (+ 3123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:36815: BlockingRecv error: recv got EOF from 127.28.210.65:36815 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
Aug 06 12:24:18 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:18 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:18.771400 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:18.765881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:48249)
0806 12:24:18.766166 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:18.766181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:18.766285 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:18.766535 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:18.766537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:18.766727 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:18.766824 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:18.766829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:18.767591 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:18.767594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:18.768196 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:18.768205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:18.768706 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:18.768710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:18.768785 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:18.769309 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:18.769322 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:18.771269 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
W20260806 12:24:18.771648 11406 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:18.766016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:48249 (local address 127.28.210.126:34519)
0806 12:24:18.766205 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:18.766209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:18.766228 (+ 19us) server_negotiation.cc:407] Connection header received
0806 12:24:18.766390 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:18.766392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:18.766437 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:18.766505 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:18.766921 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:18.767479 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:18.768304 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:18.768565 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:18.768655 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:18.771442 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:48249: BlockingRecv error: recv got EOF from 127.28.210.66:48249 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 06 12:24:18 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:18 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:18.993120 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:18.986769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:57069)
0806 12:24:18.987093 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:18.987108 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:18.987214 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:18.987494 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:18.987497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:18.987704 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:18.987815 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:18.987822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:18.988675 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:18.988678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:18.989332 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:18.989345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:18.989882 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:18.989886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:18.989972 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:18.990521 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:18.990534 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:18.992953 (+ 2419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":90,"threads_started":1}
W20260806 12:24:18.993336 11406 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:18.986973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:57069 (local address 127.28.210.126:34519)
0806 12:24:18.987124 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:18.987127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:18.987165 (+ 38us) server_negotiation.cc:407] Connection header received
0806 12:24:18.987295 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:18.987298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:18.987346 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:18.987413 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:18.987967 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:18.988566 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:18.989483 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:18.989728 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:18.989812 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:18.993119 (+ 3307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:57069: BlockingRecv error: recv got EOF from 127.28.210.67:57069 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 12:24:19 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:19 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:19.573362 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:19.567585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:43799)
0806 12:24:19.567908 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:19.567923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:19.568032 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:19.568340 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:19.568343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:19.568542 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:19.568650 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:19.568656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:19.569391 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:19.569394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:19.570046 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:19.570057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:19.570594 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:19.570597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:19.570690 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:19.571257 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:19.571270 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:19.573226 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":99,"threads_started":1}
W20260806 12:24:19.573575 11411 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:19.567747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:43799 (local address 127.28.210.126:34519)
0806 12:24:19.567989 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:19.567993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:19.568010 (+ 17us) server_negotiation.cc:407] Connection header received
0806 12:24:19.568137 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:19.568140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:19.568188 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:19.568269 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:19.568758 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:19.569294 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:19.570153 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:19.570436 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:19.570570 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:19.573435 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:43799: BlockingRecv error: recv got EOF from 127.28.210.65:43799 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
Aug 06 12:24:19 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:19 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:19.777952 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:19.772288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:50109)
0806 12:24:19.772577 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:19.772591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:19.772714 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:19.772999 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:19.773002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:19.773193 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:19.773318 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:19.773325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:19.774051 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:19.774054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:19.774634 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:19.774642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:19.775059 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:19.775062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:19.775135 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:19.775696 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:19.775708 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:19.777791 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
W20260806 12:24:19.778091 11411 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:19.772377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:50109 (local address 127.28.210.126:34519)
0806 12:24:19.772539 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:19.772544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:19.772707 (+ 163us) server_negotiation.cc:407] Connection header received
0806 12:24:19.772796 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:19.772799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:19.772850 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:19.772927 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:19.773422 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:19.773944 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:19.774720 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:19.774932 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:19.774999 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:19.777982 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:50109: BlockingRecv error: recv got EOF from 127.28.210.66:50109 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 12:24:19 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:19 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:19.999370 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:19.994003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:56555)
0806 12:24:19.994312 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:19.994324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:19.994423 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:19.994634 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:19.994636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:19.994815 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:19.994910 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:19.994915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:19.995739 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:19.995741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:19.996351 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:19.996360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:19.996757 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:19.996759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:19.996815 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:19.997287 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:19.997299 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:19.999228 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":90,"threads_started":1}
W20260806 12:24:19.999562 11411 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:19.994102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:56555 (local address 127.28.210.126:34519)
0806 12:24:19.994223 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:19.994227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:19.994391 (+ 164us) server_negotiation.cc:407] Connection header received
0806 12:24:19.994487 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:19.994490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:19.994543 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:19.994610 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:19.995048 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:19.995650 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:19.996449 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:19.996672 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:19.996748 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:19.999441 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:56555: BlockingRecv error: recv got EOF from 127.28.210.67:56555 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 06 12:24:20 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:20 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:20.591173 11415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:20.574375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:51707)
0806 12:24:20.574676 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:20.574691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:20.574795 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:20.575057 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:20.575059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:20.575253 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:20.575359 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:20.575366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:20.576299 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:20.576303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:20.576901 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:20.576914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:20.577407 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:20.577410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:20.577502 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:20.588322 (+ 10820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:20.588341 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:20.591020 (+ 2679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
W20260806 12:24:20.591418 11416 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:20.574504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:51707 (local address 127.28.210.126:34519)
0806 12:24:20.574749 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:20.574753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:20.574767 (+ 14us) server_negotiation.cc:407] Connection header received
0806 12:24:20.574892 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:20.574894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:20.574946 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:20.575027 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:20.575506 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:20.576125 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:20.577036 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:20.577276 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:20.577358 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:20.591230 (+ 13872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:51707: BlockingRecv error: recv got EOF from 127.28.210.65:51707 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
Aug 06 12:24:20 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:20 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:20.785331 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:20.778947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:34147)
0806 12:24:20.779299 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:20.779317 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:20.779446 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:20.779897 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:20.779900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:20.780312 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:20.780433 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:20.780440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:20.781248 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:20.781252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:20.781834 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:20.781844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:20.782305 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:20.782309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:20.782389 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:20.782947 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:20.782961 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:20.785186 (+ 2225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":138,"threads_started":1}
W20260806 12:24:20.785689 11416 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:20.779020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:34147 (local address 127.28.210.126:34519)
0806 12:24:20.779179 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:20.779185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:20.779428 (+ 243us) server_negotiation.cc:407] Connection header received
0806 12:24:20.779585 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:20.779589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:20.779651 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:20.779877 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:20.780551 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:20.781139 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:20.781960 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:20.782198 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:20.782286 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:20.785555 (+ 3269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:34147: BlockingRecv error: recv got EOF from 127.28.210.66:34147 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 12:24:21 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:21 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:21.007527 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:21.000471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:41455)
0806 12:24:21.000818 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:21.000836 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:21.000970 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:21.001292 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:21.001295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:21.001543 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:21.001725 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:21.001735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:21.002633 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:21.002637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:21.003273 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:21.003284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:21.003899 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:21.003903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:21.003975 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:21.004686 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:21.004702 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:21.007349 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
W20260806 12:24:21.007608 11416 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:21.000551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:41455 (local address 127.28.210.126:34519)
0806 12:24:21.000721 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:21.000726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:21.000938 (+ 212us) server_negotiation.cc:407] Connection header received
0806 12:24:21.001091 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:21.001094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:21.001158 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:21.001245 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:21.001900 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:21.002493 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:21.003425 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:21.003741 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:21.003884 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:21.007499 (+ 3615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:41455: BlockingRecv error: recv got EOF from 127.28.210.67:41455 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 12:24:21 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:21 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:21.598471 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:21.592133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:32973)
0806 12:24:21.592434 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:21.592452 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:21.592583 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:21.592901 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:21.592903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:21.593110 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:21.593216 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:21.593223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:21.594081 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:21.594085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:21.594702 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:21.594712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:21.595325 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:21.595329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:21.595401 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:21.596002 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:21.596016 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:21.598324 (+ 2308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
W20260806 12:24:21.598650 11424 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:21.592338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:32973 (local address 127.28.210.126:34519)
0806 12:24:21.592627 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:21.592631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:21.592644 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:24:21.592683 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:21.592685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:21.592731 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:21.592814 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:21.593352 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:21.593942 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:21.594923 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:21.595176 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:21.595262 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:21.598501 (+ 3239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:32973: BlockingRecv error: recv got EOF from 127.28.210.65:32973 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":85,"threads_started":1}
Aug 06 12:24:21 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:21 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:21.792328 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:21.786277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:36595)
0806 12:24:21.786602 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:21.786620 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:21.786749 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:21.787005 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:21.787008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:21.787221 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:21.787321 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:21.787328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:21.788153 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:21.788157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:21.788749 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:21.788760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:21.789245 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:21.789249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:21.789332 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:21.789929 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:21.789942 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:21.792176 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":94,"threads_started":1}
W20260806 12:24:21.792698 11424 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:21.786425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:36595 (local address 127.28.210.126:34519)
0806 12:24:21.786604 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:21.786609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:21.786707 (+ 98us) server_negotiation.cc:407] Connection header received
0806 12:24:21.786842 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:21.786845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:21.786898 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:21.786969 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:21.787453 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:21.787994 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:21.788889 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:21.789124 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:21.789222 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:21.792390 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:36595: BlockingRecv error: recv got EOF from 127.28.210.66:36595 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 12:24:22 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:22 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:22.015691 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:22.008656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:39585)
0806 12:24:22.008970 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:22.009001 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:22.009141 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:22.009447 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:22.009450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:22.009676 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:22.009803 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:22.009811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:22.010737 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:22.010740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:22.011335 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:22.011346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:22.011892 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:22.011895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:22.011970 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:22.012684 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:22.012698 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:22.015492 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
W20260806 12:24:22.015870 11424 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:22.008882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:39585 (local address 127.28.210.126:34519)
0806 12:24:22.009001 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:22.009006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:22.009076 (+ 70us) server_negotiation.cc:407] Connection header received
0806 12:24:22.009259 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:22.009262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:22.009330 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:22.009411 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:22.009925 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:22.010597 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:22.011490 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:22.011741 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:22.011851 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:22.015731 (+ 3880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:39585: BlockingRecv error: recv got EOF from 127.28.210.67:39585 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 06 12:24:22 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:22 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:22.605773 11427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:22.599613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:57357)
0806 12:24:22.600022 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:22.600036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:22.600178 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:22.600463 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:22.600466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:22.600654 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:22.600759 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:22.600765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:22.601670 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:22.601674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:22.602295 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:22.602305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:22.602786 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:22.602790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:22.602878 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:22.603429 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:22.603443 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:22.605598 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":101,"threads_started":1}
W20260806 12:24:22.605939 11428 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:22.599757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:57357 (local address 127.28.210.126:34519)
0806 12:24:22.600141 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:22.600146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:22.600167 (+ 21us) server_negotiation.cc:407] Connection header received
0806 12:24:22.600288 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:22.600290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:22.600342 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:22.600417 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:22.600859 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:22.601523 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:22.602419 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:22.602667 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:22.602760 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:22.605801 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:57357: BlockingRecv error: recv got EOF from 127.28.210.65:57357 (error 108)
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":93,"threads_started":1}
Aug 06 12:24:22 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:22 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:22.800016 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:22.793771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:34097)
0806 12:24:22.794195 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:22.794215 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:22.794347 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:22.794615 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:22.794618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:22.794864 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:22.794969 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:22.794975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:22.795800 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:22.795805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:22.796454 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:22.796466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:22.796957 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:22.796960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:22.797058 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:22.797656 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:22.797670 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:22.799838 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":144,"threads_started":1}
W20260806 12:24:22.800371 11428 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:22.793861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:34097 (local address 127.28.210.126:34519)
0806 12:24:22.794023 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:22.794028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:22.794306 (+ 278us) server_negotiation.cc:407] Connection header received
0806 12:24:22.794442 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:22.794445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:22.794501 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:22.794574 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:22.795111 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:22.795679 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:22.796589 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:22.796840 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:22.796936 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:22.800228 (+ 3292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:34097: BlockingRecv error: recv got EOF from 127.28.210.66:34097 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 12:24:23 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:23 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.67@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:23.022913 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:23.017155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.67:60467)
0806 12:24:23.017511 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:23.017526 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:23.017647 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:23.017886 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:23.017889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:23.018112 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:23.018213 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:23.018219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:23.018978 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:23.018982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:23.019571 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:23.019579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:23.020045 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:23.020048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:23.020152 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:23.020687 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:23.020699 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:23.022714 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
W20260806 12:24:23.023025 11428 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:23.017269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:60467 (local address 127.28.210.126:34519)
0806 12:24:23.017479 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:23.017484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:23.017595 (+ 111us) server_negotiation.cc:407] Connection header received
0806 12:24:23.017734 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:23.017737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:23.017793 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:23.017863 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:23.018338 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:23.018872 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:23.019690 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:23.019920 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:23.020032 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:23.022901 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.67:60467: BlockingRecv error: recv got EOF from 127.28.210.67:60467 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 06 12:24:23 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:23 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.65@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:23.613296 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:23.607060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.65:58391)
0806 12:24:23.607521 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:23.607535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:23.607658 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:23.607962 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:23.607965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:23.608226 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:23.608365 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:23.608373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:23.609198 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:23.609201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:23.609814 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:23.609823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:23.610369 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:23.610373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:23.610481 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:23.611034 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:23.611047 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:23.613151 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":113,"threads_started":1}
W20260806 12:24:23.613514 11434 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:23.607249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:58391 (local address 127.28.210.126:34519)
0806 12:24:23.607515 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:23.607519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:23.607650 (+ 131us) server_negotiation.cc:407] Connection header received
0806 12:24:23.607744 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:23.607747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:23.607798 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:23.607885 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:23.608501 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:23.609079 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:23.609976 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:23.610224 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:23.610308 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:23.613333 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.65:58391: BlockingRecv error: recv got EOF from 127.28.210.65:58391 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":65,"threads_started":1}
Aug 06 12:24:23 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 12:24:23 dist-test-slave-lgr5 krb5kdc[9925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.210.66@KRBTEST.COM for oryx/127.28.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 12:24:23.807184 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:23.801300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34519 (local address 127.28.210.66:36499)
0806 12:24:23.801659 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:23.801675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:23.801800 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:23.802037 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:23.802039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:23.802220 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:23.802316 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:23.802321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:23.803130 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:23.803133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:23.803736 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:23.803746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:23.804317 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:23.804321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:23.804410 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:23.804975 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:23.804992 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:23.807004 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.210.126:34519: Server oryx/127.28.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
W20260806 12:24:23.807265 11434 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:23.801345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:36499 (local address 127.28.210.126:34519)
0806 12:24:23.801518 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:23.801523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:23.801808 (+ 285us) server_negotiation.cc:407] Connection header received
0806 12:24:23.801883 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:23.801885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:23.801936 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:23.802007 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:23.802457 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:23.803003 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:23.803886 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:23.804168 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:23.804287 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:23.807161 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.210.66:36499: BlockingRecv error: recv got EOF from 127.28.210.66:36499 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 12:24:23.901685 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 10746
I20260806 12:24:23.919767 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 10989
I20260806 12:24:23.935099 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 11174
I20260806 12:24:23.949926 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 10650
2026-08-06T12:24:23Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18560 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T12:24:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:24:23Z Disabled control of system clock
I20260806 12:24:23.992439 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:46569
--webserver_interface=127.28.210.126
--webserver_port=0
--builtin_ntp_servers=127.28.210.84:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:46569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 12:24:24.117606 11440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:24.117952 11440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:24.118024 11440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:24.122561 11440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:24:24.122658 11440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:24.122673 11440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:24.122689 11440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:24:24.122704 11440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:24:24.128068 11440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:46569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:46569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:24.129489 11440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:24.131376 11440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:24:24.141402 11440 server_base.cc:1034] running on GCE node
W20260806 12:24:24.141335 11447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:24.141346 11446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:24.141340 11450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:24.142434 11440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:24.143932 11440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:24.145136 11440 hybrid_clock.cc:648] HybridClock initialized: now 1786019064145119 us; error 37 us; skew 500 ppm
I20260806 12:24:24.147482 11440 webserver.cc:460] Webserver started at http://127.28.210.126:44599/ using document root <none> and password file <none>
I20260806 12:24:24.148216 11440 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:24.148304 11440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:24.148547 11440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:24.151057 11440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "2792dc8ce46649e4ad1dfc368df020a5"
format_stamp: "Formatted at 2026-08-06 12:24:24 on dist-test-slave-lgr5"
server_key: "b6d4a7e7f6e131524abe1ba5f1ba7a5d"
server_key_iv: "ddaeccff7a0e6668ec0d143f6313399b"
server_key_version: "encryptionkey@0"
I20260806 12:24:24.151741 11440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "2792dc8ce46649e4ad1dfc368df020a5"
format_stamp: "Formatted at 2026-08-06 12:24:24 on dist-test-slave-lgr5"
server_key: "b6d4a7e7f6e131524abe1ba5f1ba7a5d"
server_key_iv: "ddaeccff7a0e6668ec0d143f6313399b"
server_key_version: "encryptionkey@0"
I20260806 12:24:24.156371 11440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 12:24:24.159425 11460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:24.161265 11440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:24:24.161451 11440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "2792dc8ce46649e4ad1dfc368df020a5"
format_stamp: "Formatted at 2026-08-06 12:24:24 on dist-test-slave-lgr5"
server_key: "b6d4a7e7f6e131524abe1ba5f1ba7a5d"
server_key_iv: "ddaeccff7a0e6668ec0d143f6313399b"
server_key_version: "encryptionkey@0"
I20260806 12:24:24.161623 11440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:24.174381 11440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:24.175297 11440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:24.175529 11440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:24.184741 11440 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:46569
I20260806 12:24:24.184763 11536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:46569 every 8 connection(s)
I20260806 12:24:24.185988 11440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:24:24.189958 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 11440
I20260806 12:24:24.190114 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:24:24.190404 11540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:24.190474 29513 external_mini_cluster.cc:1428] Setting key 9cfe8dcddccb1b786094318fdb905077
I20260806 12:24:24.198804 11540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5: Bootstrap starting.
I20260806 12:24:24.201342 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:24.192167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47124 (local address 127.28.210.126:46569)
0806 12:24:24.192987 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:24.193005 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:24.193065 (+ 60us) server_negotiation.cc:407] Connection header received
0806 12:24:24.193772 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:24.193800 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:24.194283 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:24.194692 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:24.195246 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.196494 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.197427 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.197974 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.198152 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:24.199212 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:24.199218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:24.199239 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:24:24.199744 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:24.199810 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:24:24.199834 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:24.199934 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:24.200145 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 12:24:24.200454 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":188,"threads_started":1}
I20260806 12:24:24.201749 11540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:24.202674 11540 log.cc:824] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5: Log is configured to *not* fsync() on all Append() calls
I20260806 12:24:24.205174 11540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5: No bootstrap required, opened a new log
I20260806 12:24:24.209429 11540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2792dc8ce46649e4ad1dfc368df020a5" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46569 } }
I20260806 12:24:24.209731 11540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:24.209770 11540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2792dc8ce46649e4ad1dfc368df020a5, State: Initialized, Role: FOLLOWER
I20260806 12:24:24.210203 11540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2792dc8ce46649e4ad1dfc368df020a5" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46569 } }
I20260806 12:24:24.210340 11540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:24:24.210384 11540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:24:24.210459 11540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:24.211503 11540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2792dc8ce46649e4ad1dfc368df020a5" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46569 } }
I20260806 12:24:24.211869 11540 leader_election.cc:302] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2792dc8ce46649e4ad1dfc368df020a5; no voters:
I20260806 12:24:24.212246 11540 leader_election.cc:288] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:24:24.212414 11547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:24:24.212698 11547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [term 1 LEADER]: Becoming Leader. State: Replica: 2792dc8ce46649e4ad1dfc368df020a5, State: Running, Role: LEADER
I20260806 12:24:24.213043 11547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2792dc8ce46649e4ad1dfc368df020a5" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46569 } }
I20260806 12:24:24.213646 11540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:24:24.215137 11549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2792dc8ce46649e4ad1dfc368df020a5. Latest consensus state: current_term: 1 leader_uuid: "2792dc8ce46649e4ad1dfc368df020a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2792dc8ce46649e4ad1dfc368df020a5" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46569 } } }
I20260806 12:24:24.215193 11548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2792dc8ce46649e4ad1dfc368df020a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2792dc8ce46649e4ad1dfc368df020a5" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46569 } } }
I20260806 12:24:24.215284 11549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [sys.catalog]: This master's current role is: LEADER
I20260806 12:24:24.215310 11548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5 [sys.catalog]: This master's current role is: LEADER
I20260806 12:24:24.215806 11558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:24:24.219457 11558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:24:24.226619 11558 catalog_manager.cc:1347] Generated new cluster ID: 63d0706dfc634ea4b83ae146c5745fc6
I20260806 12:24:24.226723 11558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:24:24.244282 11558 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:24:24.245540 11558 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:24:24.250994 11558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2792dc8ce46649e4ad1dfc368df020a5: Generated new TSK 0
I20260806 12:24:24.251796 11558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:24:24.262154 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:46569
--builtin_ntp_servers=127.28.210.84:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 12:24:24.393257 11577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:24.393589 11577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:24.393628 11577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:24.398097 11577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:24.398190 11577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:24.398247 11577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:24:24.403520 11577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:46569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:24.404860 11577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:24.406440 11577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:24.417277 11588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:24.417308 11586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:24.417385 11577 server_base.cc:1034] running on GCE node
W20260806 12:24:24.417308 11590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:24.418109 11577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:24.418862 11577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:24.420024 11577 hybrid_clock.cc:648] HybridClock initialized: now 1786019064419995 us; error 46 us; skew 500 ppm
I20260806 12:24:24.422004 11577 webserver.cc:460] Webserver started at http://127.28.210.65:43077/ using document root <none> and password file <none>
I20260806 12:24:24.422626 11577 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:24.422696 11577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:24.422914 11577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:24.425164 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "03c8bcdc1c784289b4ff2a75c8a2b014"
format_stamp: "Formatted at 2026-08-06 12:24:24 on dist-test-slave-lgr5"
server_key: "16b9bc3e5a9bb3ade4dc2ab74d76db86"
server_key_iv: "e0f3ffce369e1aa8cfbae246bc922439"
server_key_version: "encryptionkey@0"
I20260806 12:24:24.425756 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "03c8bcdc1c784289b4ff2a75c8a2b014"
format_stamp: "Formatted at 2026-08-06 12:24:24 on dist-test-slave-lgr5"
server_key: "16b9bc3e5a9bb3ade4dc2ab74d76db86"
server_key_iv: "e0f3ffce369e1aa8cfbae246bc922439"
server_key_version: "encryptionkey@0"
I20260806 12:24:24.429737 11577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 12:24:24.432564 11600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:24.433898 11577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:24:24.434042 11577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "03c8bcdc1c784289b4ff2a75c8a2b014"
format_stamp: "Formatted at 2026-08-06 12:24:24 on dist-test-slave-lgr5"
server_key: "16b9bc3e5a9bb3ade4dc2ab74d76db86"
server_key_iv: "e0f3ffce369e1aa8cfbae246bc922439"
server_key_version: "encryptionkey@0"
I20260806 12:24:24.434167 11577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:24.450428 11577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:24.451236 11577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:24.451442 11577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:24.452118 11577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:24.453183 11577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:24.453234 11577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:24.453291 11577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:24.453325 11577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:24.463017 11577 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:43691
I20260806 12:24:24.463071 11771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:43691 every 8 connection(s)
I20260806 12:24:24.464161 11577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:24:24.471444 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 11577
I20260806 12:24:24.471585 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:24:24.471906 29513 external_mini_cluster.cc:1428] Setting key 3c93961470b19987cef6009d675cf1ac
I20260806 12:24:24.474220 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:46569
--builtin_ntp_servers=127.28.210.84:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 12:24:24.475090 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:24.466305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:39113 (local address 127.28.210.126:46569)
0806 12:24:24.466462 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:24.466466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:24.467831 (+ 1365us) server_negotiation.cc:407] Connection header received
0806 12:24:24.469403 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:24.469407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:24.469492 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:24.469568 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:24.470643 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.471236 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.472673 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.472984 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.473108 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:24.474532 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:24.474536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:24.474539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:24:24.474556 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:24.474582 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:24:24.474593 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:24.474669 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:24.474802 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 12:24:24.474917 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 12:24:24.476188 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:24.466792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:46569 (local address 127.28.210.65:39113)
0806 12:24:24.467645 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:24.467694 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:24.469137 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:24.469801 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:24.469810 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:24.469910 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:24.470478 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:24.470496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.471389 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.471393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:24.472529 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:24.472546 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.473242 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.473246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:24.473528 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:24.474205 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:24:24.474207 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:24.474256 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:24:24.474345 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:24.474672 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:24.474678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:24.474682 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 12:24:24.474786 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 12:24:24.475116 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":163,"threads_started":1}
I20260806 12:24:24.477892 11772 heartbeater.cc:347] Connected to a master server at 127.28.210.126:46569
I20260806 12:24:24.478183 11772 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:24.478844 11772 heartbeater.cc:511] Master 127.28.210.126:46569 requested a full tablet report, sending...
I20260806 12:24:24.481048 11483 ts_manager.cc:194] Registered new tserver with Master: 03c8bcdc1c784289b4ff2a75c8a2b014 (127.28.210.65:43691)
I20260806 12:24:24.483013 11483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.210.65:39113
W20260806 12:24:24.604686 11775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:24.605044 11775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:24.605087 11775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:24.609611 11775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:24.609706 11775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:24.609767 11775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:24:24.615478 11775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:46569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:24.617357 11775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:24.619094 11775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:24.631251 11781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:24.631492 11775 server_base.cc:1034] running on GCE node
W20260806 12:24:24.631251 11784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:24.631256 11782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:24.632287 11775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:24.633071 11775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:24.634289 11775 hybrid_clock.cc:648] HybridClock initialized: now 1786019064634251 us; error 50 us; skew 500 ppm
I20260806 12:24:24.637001 11775 webserver.cc:460] Webserver started at http://127.28.210.66:35389/ using document root <none> and password file <none>
I20260806 12:24:24.637806 11775 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:24.637894 11775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:24.638209 11775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:24.640815 11775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "17584b4bc14a4a5cb479629a1586bf93"
format_stamp: "Formatted at 2026-08-06 12:24:24 on dist-test-slave-lgr5"
server_key: "b32200a87f05fcdb21b9461e1010e8c7"
server_key_iv: "b627ececce8e6584ca26833a6d53b314"
server_key_version: "encryptionkey@0"
I20260806 12:24:24.641467 11775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "17584b4bc14a4a5cb479629a1586bf93"
format_stamp: "Formatted at 2026-08-06 12:24:24 on dist-test-slave-lgr5"
server_key: "b32200a87f05fcdb21b9461e1010e8c7"
server_key_iv: "b627ececce8e6584ca26833a6d53b314"
server_key_version: "encryptionkey@0"
I20260806 12:24:24.645852 11775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:24:24.648957 11791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:24.650349 11775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:24:24.650506 11775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "17584b4bc14a4a5cb479629a1586bf93"
format_stamp: "Formatted at 2026-08-06 12:24:24 on dist-test-slave-lgr5"
server_key: "b32200a87f05fcdb21b9461e1010e8c7"
server_key_iv: "b627ececce8e6584ca26833a6d53b314"
server_key_version: "encryptionkey@0"
I20260806 12:24:24.650638 11775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:24.676945 11775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:24.678225 11775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:24.678481 11775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:24.679378 11775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:24.680598 11775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:24.680665 11775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:24.680709 11775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:24.680723 11775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:24.690929 11775 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:45363
I20260806 12:24:24.690966 11948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:45363 every 8 connection(s)
I20260806 12:24:24.692060 11775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:24:24.694638 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 11775
I20260806 12:24:24.694779 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:24:24.695278 29513 external_mini_cluster.cc:1428] Setting key 99082a82552fd6f10b936c343a3ac2ed
I20260806 12:24:24.697955 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:46569
--builtin_ntp_servers=127.28.210.84:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 12:24:24.705288 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:24.694410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:45163 (local address 127.28.210.126:46569)
0806 12:24:24.694608 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:24.694613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:24.696444 (+ 1831us) server_negotiation.cc:407] Connection header received
0806 12:24:24.698331 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:24.698335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:24.698403 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:24.698494 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:24.699588 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.700186 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.702012 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.702449 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.702828 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:24.704573 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:24.704577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:24.704579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:24:24.704597 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:24.704621 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:24:24.704629 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:24.704704 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:24.705060 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 12:24:24.705140 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 12:24:24.706615 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:24.695185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:46569 (local address 127.28.210.66:45163)
0806 12:24:24.696230 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:24.696303 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:24.698031 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:24.698718 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:24.698727 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:24.698831 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:24.699417 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:24.699433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.700329 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.700332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:24.701805 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:24.701821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.702950 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.702957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:24.703234 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:24.704074 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:24:24.704113 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:24.704195 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:24:24.704320 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:24.704731 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:24.704741 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:24.704749 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 12:24:24.704928 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 12:24:24.705343 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":184,"threads_started":1}
I20260806 12:24:24.708226 11953 heartbeater.cc:347] Connected to a master server at 127.28.210.126:46569
I20260806 12:24:24.708520 11953 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:24.709225 11953 heartbeater.cc:511] Master 127.28.210.126:46569 requested a full tablet report, sending...
I20260806 12:24:24.710853 11483 ts_manager.cc:194] Registered new tserver with Master: 17584b4bc14a4a5cb479629a1586bf93 (127.28.210.66:45363)
I20260806 12:24:24.711787 11483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.210.66:45163
W20260806 12:24:24.831451 11956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:24.831785 11956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:24.831851 11956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:24.836253 11956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:24.836336 11956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:24.836417 11956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:24:24.841885 11956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:46569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:24.843142 11956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:24.844724 11956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:24.855799 11964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:24.855880 11968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:24.855880 11965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:24.856357 11956 server_base.cc:1034] running on GCE node
I20260806 12:24:24.856830 11956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:24.857530 11956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:24.858728 11956 hybrid_clock.cc:648] HybridClock initialized: now 1786019064858702 us; error 35 us; skew 500 ppm
I20260806 12:24:24.860836 11956 webserver.cc:460] Webserver started at http://127.28.210.67:43911/ using document root <none> and password file <none>
I20260806 12:24:24.861487 11956 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:24.861567 11956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:24.861804 11956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:24.864420 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "486c62a8d5f44764a8d473645c581df0"
format_stamp: "Formatted at 2026-08-06 12:24:24 on dist-test-slave-lgr5"
server_key: "58a5420fb64c0f7143907760d3f6639c"
server_key_iv: "99a68d3d62059027b1969d4448d6e9b6"
server_key_version: "encryptionkey@0"
I20260806 12:24:24.865192 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "486c62a8d5f44764a8d473645c581df0"
format_stamp: "Formatted at 2026-08-06 12:24:24 on dist-test-slave-lgr5"
server_key: "58a5420fb64c0f7143907760d3f6639c"
server_key_iv: "99a68d3d62059027b1969d4448d6e9b6"
server_key_version: "encryptionkey@0"
I20260806 12:24:24.870007 11956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 12:24:24.873278 11977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:24.875100 11956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:24:24.875299 11956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "486c62a8d5f44764a8d473645c581df0"
format_stamp: "Formatted at 2026-08-06 12:24:24 on dist-test-slave-lgr5"
server_key: "58a5420fb64c0f7143907760d3f6639c"
server_key_iv: "99a68d3d62059027b1969d4448d6e9b6"
server_key_version: "encryptionkey@0"
I20260806 12:24:24.875501 11956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:24.892149 11956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:24.892925 11956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:24.893113 11956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:24.893762 11956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:24.894799 11956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:24.894850 11956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:24.894905 11956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:24.894934 11956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:24.904203 11956 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:36673
I20260806 12:24:24.904232 12133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:36673 every 8 connection(s)
I20260806 12:24:24.905180 11956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:24:24.907649 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 11956
I20260806 12:24:24.907788 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:24:24.908121 29513 external_mini_cluster.cc:1428] Setting key 728f68259c66255b69ba5d4af9dc49b6
I20260806 12:24:24.915006 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:24.907119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:56967 (local address 127.28.210.126:46569)
0806 12:24:24.907274 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:24.907279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:24.908585 (+ 1306us) server_negotiation.cc:407] Connection header received
0806 12:24:24.910162 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:24.910166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:24.910228 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:24.910359 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:24.911338 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.911892 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.913089 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.913315 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.913406 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:24.914523 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:24.914526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:24.914529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:24:24.914542 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:24.914564 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:24:24.914571 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:24.914629 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:24.914750 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 12:24:24.914823 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 12:24:24.915894 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:24.907567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:46569 (local address 127.28.210.67:56967)
0806 12:24:24.908397 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:24.908454 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:24.909946 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:24.910542 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:24.910549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:24.910638 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:24.911171 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:24.911185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.911993 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.911995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:24.912960 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:24.912971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:24.913460 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:24.913462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:24.913612 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:24.914228 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:24:24.914230 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:24.914288 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:24:24.914365 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:24.914639 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:24.914644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:24.914649 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 12:24:24.914748 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 12:24:24.915050 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":117,"threads_started":1}
I20260806 12:24:24.917133 12134 heartbeater.cc:347] Connected to a master server at 127.28.210.126:46569
I20260806 12:24:24.917411 12134 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:24.918007 12134 heartbeater.cc:511] Master 127.28.210.126:46569 requested a full tablet report, sending...
I20260806 12:24:24.919138 11483 ts_manager.cc:194] Registered new tserver with Master: 486c62a8d5f44764a8d473645c581df0 (127.28.210.67:36673)
I20260806 12:24:24.919749 11483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.210.67:56967
I20260806 12:24:24.922253 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 12:24:24.925191 11544 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:24.924260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.28.210.126:46569)
0806 12:24:24.924390 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:24.924394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:24.924564 (+ 170us) server_negotiation.cc:407] Connection header received
0806 12:24:24.924716 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:24.924719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:24.924767 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:24.924842 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:24.925043 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47132: BlockingRecv error: recv got EOF from 127.0.0.1:47132 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 12:24:24.925521 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 11577
I20260806 12:24:24.946422 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 11775
I20260806 12:24:24.957077 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 11956
I20260806 12:24:24.973434 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 11440
2026-08-06T12:24:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1016 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T12:24:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:24:25Z Disabled control of system clock
I20260806 12:24:25.008720 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:33957
--webserver_interface=127.28.210.126
--webserver_port=0
--builtin_ntp_servers=127.28.210.84:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:33957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 12:24:25.137775 12156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:25.138159 12156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:25.138206 12156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:25.142817 12156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:24:25.142918 12156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:25.142933 12156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:25.142951 12156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:24:25.142964 12156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:24:25.148655 12156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:33957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:33957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:25.149956 12156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:25.151471 12156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:25.161461 12174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:25.161446 12169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:25.161661 12156 server_base.cc:1034] running on GCE node
W20260806 12:24:25.161443 12178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:25.162457 12156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:25.163740 12156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:25.164947 12156 hybrid_clock.cc:648] HybridClock initialized: now 1786019065164928 us; error 35 us; skew 500 ppm
I20260806 12:24:25.167212 12156 webserver.cc:460] Webserver started at http://127.28.210.126:36901/ using document root <none> and password file <none>
I20260806 12:24:25.167884 12156 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:25.167955 12156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:25.168221 12156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:25.170467 12156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "0eb3e5e6012f478394e2ea301e8de58a"
format_stamp: "Formatted at 2026-08-06 12:24:25 on dist-test-slave-lgr5"
server_key: "a917aa9ee8be4e5e05d5d413c8568614"
server_key_iv: "c3737f3bb26047cec1aca34ceaf1e81a"
server_key_version: "encryptionkey@0"
I20260806 12:24:25.171065 12156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "0eb3e5e6012f478394e2ea301e8de58a"
format_stamp: "Formatted at 2026-08-06 12:24:25 on dist-test-slave-lgr5"
server_key: "a917aa9ee8be4e5e05d5d413c8568614"
server_key_iv: "c3737f3bb26047cec1aca34ceaf1e81a"
server_key_version: "encryptionkey@0"
I20260806 12:24:25.175335 12156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 12:24:25.178782 12184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:25.180521 12156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 12:24:25.180711 12156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "0eb3e5e6012f478394e2ea301e8de58a"
format_stamp: "Formatted at 2026-08-06 12:24:25 on dist-test-slave-lgr5"
server_key: "a917aa9ee8be4e5e05d5d413c8568614"
server_key_iv: "c3737f3bb26047cec1aca34ceaf1e81a"
server_key_version: "encryptionkey@0"
I20260806 12:24:25.180856 12156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:25.186323 12156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:25.187124 12156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:25.187319 12156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:25.195272 12156 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:33957
I20260806 12:24:25.195295 12257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:33957 every 8 connection(s)
I20260806 12:24:25.196621 12156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:24:25.201426 12260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:25.206599 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 12156
I20260806 12:24:25.206760 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:24:25.207103 29513 external_mini_cluster.cc:1428] Setting key 833d80b4c29464742ffffe39e27cac3e
I20260806 12:24:25.210279 12260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a: Bootstrap starting.
I20260806 12:24:25.213526 12260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:25.214432 12260 log.cc:824] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a: Log is configured to *not* fsync() on all Append() calls
I20260806 12:24:25.214506 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:25.209245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.28.210.126:33957)
0806 12:24:25.210059 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:25.210078 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:25.210123 (+ 45us) server_negotiation.cc:407] Connection header received
0806 12:24:25.210959 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:25.210983 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:25.211441 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:25.211798 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:25.212891 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:25.212897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:25.212911 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:24:25.212924 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:25.213000 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:24:25.213018 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:25.213120 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:25.213250 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 12:24:25.213597 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":148,"threads_started":1}
I20260806 12:24:25.217355 12260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a: No bootstrap required, opened a new log
I20260806 12:24:25.221817 12260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb3e5e6012f478394e2ea301e8de58a" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 33957 } }
I20260806 12:24:25.222162 12260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:25.222200 12260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb3e5e6012f478394e2ea301e8de58a, State: Initialized, Role: FOLLOWER
I20260806 12:24:25.222711 12260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb3e5e6012f478394e2ea301e8de58a" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 33957 } }
I20260806 12:24:25.222867 12260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:24:25.222939 12260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:24:25.223047 12260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:25.224216 12260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb3e5e6012f478394e2ea301e8de58a" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 33957 } }
I20260806 12:24:25.224640 12260 leader_election.cc:302] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eb3e5e6012f478394e2ea301e8de58a; no voters:
I20260806 12:24:25.225098 12260 leader_election.cc:288] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:24:25.225276 12268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:24:25.225553 12268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [term 1 LEADER]: Becoming Leader. State: Replica: 0eb3e5e6012f478394e2ea301e8de58a, State: Running, Role: LEADER
I20260806 12:24:25.225979 12268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb3e5e6012f478394e2ea301e8de58a" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 33957 } }
I20260806 12:24:25.226934 12260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:24:25.227936 12270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eb3e5e6012f478394e2ea301e8de58a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb3e5e6012f478394e2ea301e8de58a" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 33957 } } }
I20260806 12:24:25.228308 12270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [sys.catalog]: This master's current role is: LEADER
I20260806 12:24:25.228338 12271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eb3e5e6012f478394e2ea301e8de58a. Latest consensus state: current_term: 1 leader_uuid: "0eb3e5e6012f478394e2ea301e8de58a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb3e5e6012f478394e2ea301e8de58a" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 33957 } } }
I20260806 12:24:25.228480 12271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a [sys.catalog]: This master's current role is: LEADER
I20260806 12:24:25.230270 12279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:24:25.233906 12279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:24:25.240473 12279 catalog_manager.cc:1347] Generated new cluster ID: 75c02f40f4724b9684101f4081069d76
I20260806 12:24:25.240586 12279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:24:25.255116 12279 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:24:25.255517 12279 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:24:25.268009 12279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eb3e5e6012f478394e2ea301e8de58a: Generated new TSK 0
I20260806 12:24:25.268960 12279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:24:25.279533 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:33957
--builtin_ntp_servers=127.28.210.84:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 12:24:25.409458 12293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:25.409801 12293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:25.409844 12293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:25.414400 12293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:25.414490 12293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:25.414645 12293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:24:25.420377 12293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:33957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:25.421732 12293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:25.423333 12293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:25.434624 12303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:25.434672 12299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:25.434697 12300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:25.435001 12293 server_base.cc:1034] running on GCE node
I20260806 12:24:25.435490 12293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:25.436235 12293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:25.437397 12293 hybrid_clock.cc:648] HybridClock initialized: now 1786019065437376 us; error 33 us; skew 500 ppm
I20260806 12:24:25.439699 12293 webserver.cc:460] Webserver started at http://127.28.210.65:45543/ using document root <none> and password file <none>
I20260806 12:24:25.440348 12293 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:25.440418 12293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:25.440618 12293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:25.442782 12293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "77f60fd91fab4dedb904e0b217f56980"
format_stamp: "Formatted at 2026-08-06 12:24:25 on dist-test-slave-lgr5"
server_key: "b7637fc5a7f0313b89c83fd6cfc2ff74"
server_key_iv: "2b23a85d10d7e478c22e041145bfb098"
server_key_version: "encryptionkey@0"
I20260806 12:24:25.443368 12293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "77f60fd91fab4dedb904e0b217f56980"
format_stamp: "Formatted at 2026-08-06 12:24:25 on dist-test-slave-lgr5"
server_key: "b7637fc5a7f0313b89c83fd6cfc2ff74"
server_key_iv: "2b23a85d10d7e478c22e041145bfb098"
server_key_version: "encryptionkey@0"
I20260806 12:24:25.447365 12293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 12:24:25.450340 12311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:25.451683 12293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 12:24:25.451829 12293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "77f60fd91fab4dedb904e0b217f56980"
format_stamp: "Formatted at 2026-08-06 12:24:25 on dist-test-slave-lgr5"
server_key: "b7637fc5a7f0313b89c83fd6cfc2ff74"
server_key_iv: "2b23a85d10d7e478c22e041145bfb098"
server_key_version: "encryptionkey@0"
I20260806 12:24:25.451962 12293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:25.456923 12293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:25.457620 12293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:25.457798 12293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:25.458427 12293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:25.459453 12293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:25.459502 12293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:25.459558 12293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:25.459585 12293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:25.470299 12293 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:35241
I20260806 12:24:25.470345 12458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:35241 every 8 connection(s)
I20260806 12:24:25.471382 12293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:24:25.478914 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 12293
I20260806 12:24:25.479039 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:24:25.479257 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:25.473695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:42011 (local address 127.28.210.126:33957)
0806 12:24:25.473867 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:25.473871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:25.475621 (+ 1750us) server_negotiation.cc:407] Connection header received
0806 12:24:25.477149 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:25.477153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:25.477209 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:25.477307 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:25.478748 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:25.478752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:25.478754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:24:25.478768 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:25.478791 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:24:25.478800 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:25.478891 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:25.479040 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 12:24:25.479093 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 12:24:25.479359 29513 external_mini_cluster.cc:1428] Setting key 9d4955ef8dda1b11a3e215fce5e8d55e
I20260806 12:24:25.480665 12462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:25.474114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:33957 (local address 127.28.210.65:42011)
0806 12:24:25.475407 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:25.475471 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:25.476865 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:25.477553 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:25.477629 (+ 76us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:25.477764 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:25.478422 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:24:25.478424 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:25.478472 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:24:25.478566 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:25.478881 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:25.478890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:25.478896 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 12:24:25.479077 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 12:24:25.479510 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":152,"threads_started":1}
I20260806 12:24:25.481809 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:33957
--builtin_ntp_servers=127.28.210.84:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 12:24:25.482523 12459 heartbeater.cc:347] Connected to a master server at 127.28.210.126:33957
I20260806 12:24:25.482878 12459 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:25.483560 12459 heartbeater.cc:511] Master 127.28.210.126:33957 requested a full tablet report, sending...
I20260806 12:24:25.486702 12205 ts_manager.cc:194] Registered new tserver with Master: 77f60fd91fab4dedb904e0b217f56980 (127.28.210.65:35241)
W20260806 12:24:25.620481 12464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:25.620801 12464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:25.620843 12464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:25.625452 12464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:25.625600 12464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:25.625671 12464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:24:25.631583 12464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:33957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:25.632938 12464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:25.634755 12464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:25.646109 12477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:25.646099 12472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:25.646107 12470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:25.646325 12464 server_base.cc:1034] running on GCE node
I20260806 12:24:25.647061 12464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:25.647940 12464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:25.649168 12464 hybrid_clock.cc:648] HybridClock initialized: now 1786019065649142 us; error 45 us; skew 500 ppm
I20260806 12:24:25.656803 12464 webserver.cc:460] Webserver started at http://127.28.210.66:42763/ using document root <none> and password file <none>
I20260806 12:24:25.657651 12464 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:25.657745 12464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:25.657999 12464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:25.660573 12464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "f702854cf0f04ece8850bcda449bee07"
format_stamp: "Formatted at 2026-08-06 12:24:25 on dist-test-slave-lgr5"
server_key: "8a63bf434010c1544362199d99d80ea6"
server_key_iv: "2c261ba8921a77d943fa7f3bc67b5a8b"
server_key_version: "encryptionkey@0"
I20260806 12:24:25.661249 12464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "f702854cf0f04ece8850bcda449bee07"
format_stamp: "Formatted at 2026-08-06 12:24:25 on dist-test-slave-lgr5"
server_key: "8a63bf434010c1544362199d99d80ea6"
server_key_iv: "2c261ba8921a77d943fa7f3bc67b5a8b"
server_key_version: "encryptionkey@0"
I20260806 12:24:25.665992 12464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:24:25.669273 12483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:25.671258 12464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 12:24:25.671474 12464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "f702854cf0f04ece8850bcda449bee07"
format_stamp: "Formatted at 2026-08-06 12:24:25 on dist-test-slave-lgr5"
server_key: "8a63bf434010c1544362199d99d80ea6"
server_key_iv: "2c261ba8921a77d943fa7f3bc67b5a8b"
server_key_version: "encryptionkey@0"
I20260806 12:24:25.671659 12464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:25.677724 12464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:25.678505 12464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:25.678701 12464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:25.679476 12464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:25.680907 12464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:25.680979 12464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:25.681041 12464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:25.681073 12464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:25.691757 12464 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:45391
I20260806 12:24:25.691783 12646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:45391 every 8 connection(s)
I20260806 12:24:25.692917 12464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:24:25.694550 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 12464
I20260806 12:24:25.694682 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:24:25.695032 29513 external_mini_cluster.cc:1428] Setting key a04995696a3aeb7e694833b7b3f2248c
I20260806 12:24:25.697261 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:33957
--builtin_ntp_servers=127.28.210.84:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 12:24:25.700202 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:25.695078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:55127 (local address 127.28.210.126:33957)
0806 12:24:25.695241 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:25.695246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:25.696659 (+ 1413us) server_negotiation.cc:407] Connection header received
0806 12:24:25.698199 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:25.698203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:25.698255 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:25.698335 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:25.699656 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:25.699661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:25.699664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:24:25.699680 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:25.699708 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:24:25.699718 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:25.699827 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:25.699931 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 12:24:25.699987 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 12:24:25.701239 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:25.695590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:33957 (local address 127.28.210.66:55127)
0806 12:24:25.696500 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:25.696562 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:25.697948 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:25.698539 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:25.698548 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:25.698656 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:25.699327 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:24:25.699329 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:25.699383 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:24:25.699469 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:25.699806 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:25.699811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:25.699816 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 12:24:25.699974 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 12:24:25.700292 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":146,"threads_started":1}
I20260806 12:24:25.702666 12647 heartbeater.cc:347] Connected to a master server at 127.28.210.126:33957
I20260806 12:24:25.702929 12647 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:25.703423 12647 heartbeater.cc:511] Master 127.28.210.126:33957 requested a full tablet report, sending...
I20260806 12:24:25.704671 12205 ts_manager.cc:194] Registered new tserver with Master: f702854cf0f04ece8850bcda449bee07 (127.28.210.66:45391)
W20260806 12:24:25.822649 12650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:25.822981 12650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:25.823025 12650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:25.827751 12650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:25.827845 12650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:25.827934 12650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:24:25.834043 12650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:33957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:25.835533 12650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:25.837195 12650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:24:25.848254 12650 server_base.cc:1034] running on GCE node
W20260806 12:24:25.848239 12660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:25.848234 12662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:25.848551 12657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:25.848984 12650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:25.849701 12650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:25.850876 12650 hybrid_clock.cc:648] HybridClock initialized: now 1786019065850857 us; error 34 us; skew 500 ppm
I20260806 12:24:25.852970 12650 webserver.cc:460] Webserver started at http://127.28.210.67:42319/ using document root <none> and password file <none>
I20260806 12:24:25.853611 12650 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:25.853685 12650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:25.853909 12650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:25.856307 12650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "fbf5c3e26fc8441a80dc34b27cfd59a0"
format_stamp: "Formatted at 2026-08-06 12:24:25 on dist-test-slave-lgr5"
server_key: "02d175574cf63d6cd2e922f021ade191"
server_key_iv: "446cf7ad6dd89d72cac52148b486ec64"
server_key_version: "encryptionkey@0"
I20260806 12:24:25.856917 12650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "fbf5c3e26fc8441a80dc34b27cfd59a0"
format_stamp: "Formatted at 2026-08-06 12:24:25 on dist-test-slave-lgr5"
server_key: "02d175574cf63d6cd2e922f021ade191"
server_key_iv: "446cf7ad6dd89d72cac52148b486ec64"
server_key_version: "encryptionkey@0"
I20260806 12:24:25.861104 12650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:24:25.864063 12670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:25.865526 12650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 12:24:25.865662 12650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "fbf5c3e26fc8441a80dc34b27cfd59a0"
format_stamp: "Formatted at 2026-08-06 12:24:25 on dist-test-slave-lgr5"
server_key: "02d175574cf63d6cd2e922f021ade191"
server_key_iv: "446cf7ad6dd89d72cac52148b486ec64"
server_key_version: "encryptionkey@0"
I20260806 12:24:25.865811 12650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:25.870998 12650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:25.871785 12650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:25.871984 12650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:25.872725 12650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:25.873891 12650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:25.873947 12650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:25.874009 12650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:25.874037 12650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:25.884307 12650 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:36965
I20260806 12:24:25.884354 12821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:36965 every 8 connection(s)
I20260806 12:24:25.885572 12650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:24:25.886565 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 12650
I20260806 12:24:25.886693 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:24:25.887045 29513 external_mini_cluster.cc:1428] Setting key 28fb5f7d66dc1746f8c308da0b87cbbb
I20260806 12:24:25.892956 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:25.888021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:40461 (local address 127.28.210.126:33957)
0806 12:24:25.888181 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:25.888186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:25.889819 (+ 1633us) server_negotiation.cc:407] Connection header received
0806 12:24:25.891190 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:25.891192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:25.891232 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:25.891303 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:25.892560 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:25.892563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:25.892565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:24:25.892577 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:25.892600 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:24:25.892608 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:25.892689 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:25.892793 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 12:24:25.892835 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 12:24:25.893945 12824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:25.888686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:33957 (local address 127.28.210.67:40461)
0806 12:24:25.889647 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:25.889712 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:25.890986 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:25.891466 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:25.891490 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:25.891589 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:25.892235 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:24:25.892237 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:25.892290 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:24:25.892372 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:25.892680 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:25.892686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:25.892690 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 12:24:25.892833 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 12:24:25.893053 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":179,"threads_started":1}
I20260806 12:24:25.895292 12822 heartbeater.cc:347] Connected to a master server at 127.28.210.126:33957
I20260806 12:24:25.895645 12822 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:25.896258 12822 heartbeater.cc:511] Master 127.28.210.126:33957 requested a full tablet report, sending...
I20260806 12:24:25.897737 12205 ts_manager.cc:194] Registered new tserver with Master: fbf5c3e26fc8441a80dc34b27cfd59a0 (127.28.210.67:36965)
I20260806 12:24:25.902040 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 12:24:25.905259 12265 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:24:25.904253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.28.210.126:33957)
0806 12:24:25.904390 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:25.904394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:25.904598 (+ 204us) server_negotiation.cc:407] Connection header received
0806 12:24:25.904733 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:25.904736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:25.904780 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:25.904848 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:25.905112 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54498: BlockingRecv error: recv got EOF from 127.0.0.1:54498 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 12:24:25.905589 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 12293
I20260806 12:24:25.926010 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 12464
I20260806 12:24:25.943966 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 12650
I20260806 12:24:25.976979 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 12156
2026-08-06T12:24:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (1016 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:24:26 dist-test-slave-lgr5 krb5kdc[12837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:24:26 dist-test-slave-lgr5 krb5kdc[12837](info): set up 2 sockets
Aug 06 12:24:26 dist-test-slave-lgr5 krb5kdc[12837](info): commencing operation
krb5kdc: starting...
W20260806 12:24:27.188565 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:24:27 dist-test-slave-lgr5 krb5kdc[12837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:24:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:24:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:24:27.340945 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:41035
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:36893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:41035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:27.474390 12853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:27.474742 12853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:27.474805 12853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:27.479209 12853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:24:27.479295 12853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:27.479321 12853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:27.479343 12853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:24:27.479365 12853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:24:27.485357 12853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:36893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:41035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:41035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:27.486668 12853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:27.487862 12853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:27.497776 12860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:27.497872 12853 server_base.cc:1034] running on GCE node
W20260806 12:24:27.497785 12859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:27.497776 12862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:27.498847 12853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:27.500589 12853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:27.501819 12853 hybrid_clock.cc:648] HybridClock initialized: now 1786019067501796 us; error 56 us; skew 500 ppm
Aug 06 12:24:27 dist-test-slave-lgr5 krb5kdc[12837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019067, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:27.506207 12853 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:24:27.507685 12853 webserver.cc:460] Webserver started at http://127.28.210.126:38881/ using document root <none> and password file <none>
I20260806 12:24:27.508525 12853 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:27.508626 12853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:27.508895 12853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:27.511562 12853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "2cad373ae4e14544abff7a2b2a7fa4f3"
format_stamp: "Formatted at 2026-08-06 12:24:27 on dist-test-slave-lgr5"
server_key: "97cd955d56b4dc47ab8680d51c18f9ee"
server_key_iv: "181978560ae895cc156712211ec1721d"
server_key_version: "encryptionkey@0"
I20260806 12:24:27.512432 12853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "2cad373ae4e14544abff7a2b2a7fa4f3"
format_stamp: "Formatted at 2026-08-06 12:24:27 on dist-test-slave-lgr5"
server_key: "97cd955d56b4dc47ab8680d51c18f9ee"
server_key_iv: "181978560ae895cc156712211ec1721d"
server_key_version: "encryptionkey@0"
I20260806 12:24:27.517524 12853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 12:24:27.520622 12869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:27.522159 12853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:24:27.522327 12853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "2cad373ae4e14544abff7a2b2a7fa4f3"
format_stamp: "Formatted at 2026-08-06 12:24:27 on dist-test-slave-lgr5"
server_key: "97cd955d56b4dc47ab8680d51c18f9ee"
server_key_iv: "181978560ae895cc156712211ec1721d"
server_key_version: "encryptionkey@0"
I20260806 12:24:27.522462 12853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:27.555831 12853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:27.559890 12853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:27.560189 12853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:27.569548 12853 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:41035
I20260806 12:24:27.570210 12941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:41035 every 8 connection(s)
I20260806 12:24:27.570801 12853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:24:27.574323 12942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:27.580770 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 12853
I20260806 12:24:27.580914 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:24:27.581194 29513 external_mini_cluster.cc:1428] Setting key bde7bf777c9ef66d81acaaff3632d3c4
I20260806 12:24:27.581563 12942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3: Bootstrap starting.
I20260806 12:24:27.585465 12942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:27.586546 12942 log.cc:824] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3: Log is configured to *not* fsync() on all Append() calls
I20260806 12:24:27.589241 12942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3: No bootstrap required, opened a new log
Aug 06 12:24:27 dist-test-slave-lgr5 krb5kdc[12837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:24:27.594499 12942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cad373ae4e14544abff7a2b2a7fa4f3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 41035 } }
I20260806 12:24:27.594832 12942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:27.594900 12942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cad373ae4e14544abff7a2b2a7fa4f3, State: Initialized, Role: FOLLOWER
I20260806 12:24:27.595479 12942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cad373ae4e14544abff7a2b2a7fa4f3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 41035 } }
I20260806 12:24:27.595667 12942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:24:27.595745 12942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:24:27.595860 12942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:27.597334 12942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cad373ae4e14544abff7a2b2a7fa4f3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 41035 } }
I20260806 12:24:27.597765 12942 leader_election.cc:302] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cad373ae4e14544abff7a2b2a7fa4f3; no voters:
I20260806 12:24:27.598189 12942 leader_election.cc:288] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:24:27.598336 12949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:24:27.598584 12949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [term 1 LEADER]: Becoming Leader. State: Replica: 2cad373ae4e14544abff7a2b2a7fa4f3, State: Running, Role: LEADER
I20260806 12:24:27.598963 12949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cad373ae4e14544abff7a2b2a7fa4f3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 41035 } }
I20260806 12:24:27.599812 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:27.582817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.28.210.126:41035)
0806 12:24:27.583402 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:27.583422 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:27.583466 (+ 44us) server_negotiation.cc:407] Connection header received
0806 12:24:27.584320 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:27.584345 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:27.584790 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:27.585136 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:27.586209 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:27.587362 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:27.588621 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:27.589181 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:27.589331 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:27.592963 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:27.592970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:27.592992 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:27.593048 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:27.595782 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:27.596526 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:27.596533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:27.596540 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:27.596686 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:27.596997 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:27.596999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:27.597000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:27.597519 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:27.597737 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:27.598144 (+ 407us) server_negotiation.cc:299] Negotiation successful
0806 12:24:27.598837 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":184,"threads_started":1}
I20260806 12:24:27.599820 12942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:24:27.600430 12951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cad373ae4e14544abff7a2b2a7fa4f3. Latest consensus state: current_term: 1 leader_uuid: "2cad373ae4e14544abff7a2b2a7fa4f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cad373ae4e14544abff7a2b2a7fa4f3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 41035 } } }
I20260806 12:24:27.600569 12951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [sys.catalog]: This master's current role is: LEADER
I20260806 12:24:27.600939 12952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cad373ae4e14544abff7a2b2a7fa4f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cad373ae4e14544abff7a2b2a7fa4f3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 41035 } } }
I20260806 12:24:27.601044 12952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3 [sys.catalog]: This master's current role is: LEADER
I20260806 12:24:27.601418 12954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:24:27.604941 12954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:24:27.611733 12954 catalog_manager.cc:1347] Generated new cluster ID: 0a371042ec884bc0909b329092b0f60f
I20260806 12:24:27.611853 12954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:24:27.622840 12954 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:24:27.623863 12954 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:24:27.638302 12954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3: Generated new TSK 0
I20260806 12:24:27.639083 12954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:24:27.679971 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:41035
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:36893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:27.812156 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:27.812610 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:27.812670 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:27.817803 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:27.817950 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:27.818025 12976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:24:27.824169 12976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:36893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:41035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:27.825557 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:27.826745 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:27.838467 12991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:27.838583 12976 server_base.cc:1034] running on GCE node
W20260806 12:24:27.838461 12987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:27.838502 12985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:27.839206 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:27.839974 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:27.841168 12976 hybrid_clock.cc:648] HybridClock initialized: now 1786019067841148 us; error 36 us; skew 500 ppm
Aug 06 12:24:27 dist-test-slave-lgr5 krb5kdc[12837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019067, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:27.844511 12976 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:24:27.845605 12976 webserver.cc:460] Webserver started at http://127.28.210.65:46663/ using document root <none> and password file <none>
I20260806 12:24:27.846228 12976 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:27.846302 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:27.846498 12976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:27.849063 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "39d2a66d2ee7412c92c3399cdf60413d"
format_stamp: "Formatted at 2026-08-06 12:24:27 on dist-test-slave-lgr5"
server_key: "28a4910e9a00d59ee78be15556e84412"
server_key_iv: "55d56a8fd7305a4107c59a1c189ed235"
server_key_version: "encryptionkey@0"
I20260806 12:24:27.849694 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "39d2a66d2ee7412c92c3399cdf60413d"
format_stamp: "Formatted at 2026-08-06 12:24:27 on dist-test-slave-lgr5"
server_key: "28a4910e9a00d59ee78be15556e84412"
server_key_iv: "55d56a8fd7305a4107c59a1c189ed235"
server_key_version: "encryptionkey@0"
I20260806 12:24:27.854317 12976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:24:27.857537 12999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:27.859035 12976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 12:24:27.859189 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "39d2a66d2ee7412c92c3399cdf60413d"
format_stamp: "Formatted at 2026-08-06 12:24:27 on dist-test-slave-lgr5"
server_key: "28a4910e9a00d59ee78be15556e84412"
server_key_iv: "55d56a8fd7305a4107c59a1c189ed235"
server_key_version: "encryptionkey@0"
I20260806 12:24:27.859320 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:27.882350 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:27.886734 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:27.887022 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:27.887804 12976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:27.888942 12976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:27.889011 12976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:27.889076 12976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:27.889106 12976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:27.899566 12976 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:33573
I20260806 12:24:27.899616 13155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:33573 every 8 connection(s)
I20260806 12:24:27.900722 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:24:27.909826 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 12976
I20260806 12:24:27.909960 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:24:27.910239 29513 external_mini_cluster.cc:1428] Setting key 028ebb24b02affb4cda1cb7f7cc26e38
Aug 06 12:24:27 dist-test-slave-lgr5 krb5kdc[12837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019067, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:24:27.919330 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:27.903148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:57229 (local address 127.28.210.126:41035)
0806 12:24:27.903311 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:27.903315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:27.904301 (+ 986us) server_negotiation.cc:407] Connection header received
0806 12:24:27.905840 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:27.905844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:27.905912 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:27.905991 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:27.907414 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:27.907992 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:27.909275 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:27.909563 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:27.909683 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:27.912869 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:27.912873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:27.912876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:27.912910 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:27.914652 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:27.915511 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:27.915515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:27.915516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:27.915627 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:27.916167 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:27.916172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:27.916174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:27.916479 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:27.916569 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:27.918901 (+ 2332us) server_negotiation.cc:299] Negotiation successful
0806 12:24:27.919097 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 12:24:27.920639 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:27.903551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:41035 (local address 127.28.210.65:57229)
0806 12:24:27.904128 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:27.904188 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:27.905574 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:27.906215 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:27.906225 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:27.906653 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:27.907227 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:27.907243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:27.908160 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:27.908163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:27.909130 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:27.909142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:27.909789 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:27.909792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:27.910000 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:27.910691 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:27.910710 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:27.912590 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:27.914821 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:27.914830 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:27.914854 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:27.915332 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:27.915735 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:27.915738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:27.915741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:27.915997 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:27.918110 (+ 2113us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:27.918118 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:27.918496 (+ 378us) client_negotiation.cc:769] Sending connection context
0806 12:24:27.918754 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 12:24:27.919233 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":150,"threads_started":1}
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:24:27.922390 13157 heartbeater.cc:347] Connected to a master server at 127.28.210.126:41035
I20260806 12:24:27.922689 13157 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:27.923333 13157 heartbeater.cc:511] Master 127.28.210.126:41035 requested a full tablet report, sending...
I20260806 12:24:27.925379 12891 ts_manager.cc:194] Registered new tserver with Master: 39d2a66d2ee7412c92c3399cdf60413d (127.28.210.65:33573)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:24:27.927079 12891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:57229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:24:27.947682 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:41035
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:36893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:28.080286 13165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:28.080677 13165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:28.080744 13165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:28.085791 13165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:28.085901 13165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:28.085994 13165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:24:28.091905 13165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:36893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:41035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:28.093183 13165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:28.094277 13165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:28.104833 13173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:28.104833 13180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:28.104833 13178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:28.105329 13165 server_base.cc:1034] running on GCE node
I20260806 12:24:28.105804 13165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:28.106563 13165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:28.107861 13165 hybrid_clock.cc:648] HybridClock initialized: now 1786019068107781 us; error 123 us; skew 500 ppm
Aug 06 12:24:28 dist-test-slave-lgr5 krb5kdc[12837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019068, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:28.112452 13165 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:24:28.114029 13165 webserver.cc:460] Webserver started at http://127.28.210.66:44145/ using document root <none> and password file <none>
I20260806 12:24:28.114923 13165 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:28.115051 13165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:28.115474 13165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:28.118188 13165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f"
format_stamp: "Formatted at 2026-08-06 12:24:28 on dist-test-slave-lgr5"
server_key: "7979f224617038414cf1d0d18ff9b3e0"
server_key_iv: "1543a44133375f1eeabbc3e41ad83312"
server_key_version: "encryptionkey@0"
I20260806 12:24:28.118924 13165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f"
format_stamp: "Formatted at 2026-08-06 12:24:28 on dist-test-slave-lgr5"
server_key: "7979f224617038414cf1d0d18ff9b3e0"
server_key_iv: "1543a44133375f1eeabbc3e41ad83312"
server_key_version: "encryptionkey@0"
I20260806 12:24:28.123667 13165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 12:24:28.127750 13187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:28.129769 13165 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:24:28.129988 13165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f"
format_stamp: "Formatted at 2026-08-06 12:24:28 on dist-test-slave-lgr5"
server_key: "7979f224617038414cf1d0d18ff9b3e0"
server_key_iv: "1543a44133375f1eeabbc3e41ad83312"
server_key_version: "encryptionkey@0"
I20260806 12:24:28.130146 13165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:28.168272 13165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:28.173594 13165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:28.173892 13165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:28.174710 13165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:28.175915 13165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:28.175973 13165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:28.176038 13165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:28.176070 13165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:28.187955 13165 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:42969
I20260806 12:24:28.188009 13344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:42969 every 8 connection(s)
I20260806 12:24:28.189157 13165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:24:28.198335 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 13165
I20260806 12:24:28.198472 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:24:28.198791 29513 external_mini_cluster.cc:1428] Setting key 5353d80e4b5a126b66dbfafba5d399ca
Aug 06 12:24:28 dist-test-slave-lgr5 krb5kdc[12837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019068, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:24:28.205343 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.191468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:34183 (local address 127.28.210.126:41035)
0806 12:24:28.191619 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:28.191624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:28.192703 (+ 1079us) server_negotiation.cc:407] Connection header received
0806 12:24:28.194292 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:28.194296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:28.194376 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:28.194454 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:28.195920 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.196539 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.197832 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.198071 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.198169 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.201362 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:28.201366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:28.201368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:28.201399 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:28.202883 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:28.203518 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:28.203522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:28.203524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:28.203610 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:28.203985 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:28.203988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:28.203989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:28.204251 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:28.204350 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:28.204993 (+ 643us) server_negotiation.cc:299] Negotiation successful
0806 12:24:28.205156 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 12:24:28.206403 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.191893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:41035 (local address 127.28.210.66:34183)
0806 12:24:28.192521 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:28.192574 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:28.194025 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:28.194692 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:28.194700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:28.195162 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:28.195744 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.195760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.196683 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.196686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.197688 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.197700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.198242 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.198247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.198439 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.199186 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:28.199208 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:28.201151 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:28.203043 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:28.203054 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:28.203078 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:28.203406 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:28.203704 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:28.203707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:28.203710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:28.203880 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:28.204362 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:28.204372 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:28.204698 (+ 326us) client_negotiation.cc:769] Sending connection context
0806 12:24:28.204917 (+ 219us) client_negotiation.cc:240] Negotiation successful
0806 12:24:28.205327 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":186,"threads_started":1}
I20260806 12:24:28.208182 13345 heartbeater.cc:347] Connected to a master server at 127.28.210.126:41035
I20260806 12:24:28.208526 13345 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:28.209184 13345 heartbeater.cc:511] Master 127.28.210.126:41035 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
I20260806 12:24:28.211053 12891 ts_manager.cc:194] Registered new tserver with Master: 8dc0a4eac7bc4b3690ff6e813d1fce0f (127.28.210.66:42969)
I20260806 12:24:28.211757 12891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:34183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:24:28.235734 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:41035
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:36893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:24:28.373950 13355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:24:28.374290 13355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:24:28.374356 13355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:24:28.378737 13355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:24:28.378849 13355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:24:28.378940 13355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:24:28.385227 13355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:36893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:41035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:24:28.386754 13355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:24:28.388060 13355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:24:28.399735 13363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:28.399735 13362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:24:28.399737 13368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:24:28.400149 13355 server_base.cc:1034] running on GCE node
I20260806 12:24:28.400718 13355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:24:28.401629 13355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:24:28.402810 13355 hybrid_clock.cc:648] HybridClock initialized: now 1786019068402780 us; error 48 us; skew 500 ppm
Aug 06 12:24:28 dist-test-slave-lgr5 krb5kdc[12837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019068, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:24:28.406256 13355 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:24:28.407464 13355 webserver.cc:460] Webserver started at http://127.28.210.67:43769/ using document root <none> and password file <none>
I20260806 12:24:28.408308 13355 fs_manager.cc:359] Metadata directory not provided
I20260806 12:24:28.408403 13355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:24:28.408674 13355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:24:28.411521 13355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "2d93ce3a3163436085db915f54644e6a"
format_stamp: "Formatted at 2026-08-06 12:24:28 on dist-test-slave-lgr5"
server_key: "452450abe5860212d0eff6d7cbd9ab61"
server_key_iv: "272ff3564c77df80fee807dcd7a04e31"
server_key_version: "encryptionkey@0"
I20260806 12:24:28.412359 13355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "2d93ce3a3163436085db915f54644e6a"
format_stamp: "Formatted at 2026-08-06 12:24:28 on dist-test-slave-lgr5"
server_key: "452450abe5860212d0eff6d7cbd9ab61"
server_key_iv: "272ff3564c77df80fee807dcd7a04e31"
server_key_version: "encryptionkey@0"
I20260806 12:24:28.417269 13355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260806 12:24:28.420491 13378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:28.422139 13355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 12:24:28.422346 13355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "2d93ce3a3163436085db915f54644e6a"
format_stamp: "Formatted at 2026-08-06 12:24:28 on dist-test-slave-lgr5"
server_key: "452450abe5860212d0eff6d7cbd9ab61"
server_key_iv: "272ff3564c77df80fee807dcd7a04e31"
server_key_version: "encryptionkey@0"
I20260806 12:24:28.422514 13355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:24:28.447623 13355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:24:28.451738 13355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:24:28.451989 13355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:24:28.452768 13355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:24:28.454063 13355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:24:28.454136 13355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:28.454203 13355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:24:28.454238 13355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:24:28.464591 13355 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:46205
I20260806 12:24:28.464638 13545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:46205 every 8 connection(s)
I20260806 12:24:28.465839 13355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:24:28.466205 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 13355
I20260806 12:24:28.466341 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:24:28.466739 29513 external_mini_cluster.cc:1428] Setting key 6f0e7a81cfac2838fac5dcfde1f3814b
Aug 06 12:24:28 dist-test-slave-lgr5 krb5kdc[12837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019068, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:24:28.485280 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.468243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:53033 (local address 127.28.210.126:41035)
0806 12:24:28.468406 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:28.468411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:28.469683 (+ 1272us) server_negotiation.cc:407] Connection header received
0806 12:24:28.471243 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:28.471247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:28.471316 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:28.471397 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:28.472951 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.473545 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.475406 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.475709 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.475830 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.479312 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:28.479317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:28.479319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:28.479362 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:28.481553 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:28.482465 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:28.482469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:28.482470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:28.482584 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:28.483032 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:28.483036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:28.483037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:28.483526 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:28.483730 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:28.484692 (+ 962us) server_negotiation.cc:299] Negotiation successful
0806 12:24:28.484915 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 12:24:28.486459 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.468832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:41035 (local address 127.28.210.67:53033)
0806 12:24:28.469519 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:28.469578 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:28.470982 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:28.471624 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:28.471635 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:28.472147 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:24:28.472771 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.472788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.473698 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.473701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.475154 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.475170 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.475925 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.475931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.476205 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.476983 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:24:28.477012 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:24:28.479003 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:24:28.481845 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:28.481854 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:28.481892 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:28.482297 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:28.482694 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:24:28.482697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:24:28.482699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:24:28.482907 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:24:28.483785 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:24:28.483796 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:24:28.484320 (+ 524us) client_negotiation.cc:769] Sending connection context
0806 12:24:28.484602 (+ 282us) client_negotiation.cc:240] Negotiation successful
0806 12:24:28.485329 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":168,"threads_started":1}
I20260806 12:24:28.487963 13546 heartbeater.cc:347] Connected to a master server at 127.28.210.126:41035
I20260806 12:24:28.488401 13546 heartbeater.cc:464] Registering TS with master...
I20260806 12:24:28.489076 13546 heartbeater.cc:511] Master 127.28.210.126:41035 requested a full tablet report, sending...
I20260806 12:24:28.490520 12891 ts_manager.cc:194] Registered new tserver with Master: 2d93ce3a3163436085db915f54644e6a (127.28.210.67:46205)
I20260806 12:24:28.491189 12891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:53033
I20260806 12:24:28.496157 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:24:28.508352 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.498729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.28.210.126:41035)
0806 12:24:28.498943 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:28.498947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:28.498967 (+ 20us) server_negotiation.cc:407] Connection header received
0806 12:24:28.499185 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:28.499190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:28.499254 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:28.499345 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:24:28.500239 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.500865 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.502043 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.502291 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.502388 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.504588 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:24:28.504605 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:24:28.504608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:24:28.504651 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:24:28.506257 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:28.506874 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:28.506878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:28.506879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:28.506987 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:24:28.507357 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:24:28.507361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:24:28.507362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:24:28.507657 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:24:28.507782 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:28.508016 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 12:24:28.508189 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 12:24:28.512024 12891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:24:28.516839 12891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:24:28.535997 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.529461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.28.210.67:46205)
0806 12:24:28.529999 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:28.530011 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:28.530042 (+ 31us) server_negotiation.cc:407] Connection header received
0806 12:24:28.530409 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:28.530414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:28.530638 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:28.530816 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:24:28.531152 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.532211 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.533882 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.534933 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.535089 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.535389 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:28.535662 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 12:24:28.535795 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":206,"threads_started":1}
I20260806 12:24:28.535985 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.529335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:46205 (local address 127.0.0.1:44732)
0806 12:24:28.529793 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:28.529852 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:28.530008 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:28.530807 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:28.530814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:28.530842 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:24:28.531008 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.531017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.532356 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.532360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.533680 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.533693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.535200 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.535205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.535386 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.535437 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 12:24:28.535588 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 12:24:28.535795 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":151,"threads_started":1}
I20260806 12:24:28.536942 13569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.530764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:42969 (local address 127.0.0.1:56730)
0806 12:24:28.531195 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:28.531209 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:28.531304 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:28.532129 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:28.532132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:28.532153 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:24:28.532254 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.532260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.533661 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.533666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.534898 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.534914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.536576 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.536580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.536663 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.536685 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 12:24:28.536740 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 12:24:28.536817 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":113,"threads_started":1}
I20260806 12:24:28.537285 13567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.530342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37972 (local address 127.28.210.65:33573)
0806 12:24:28.530685 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:28.530692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:28.531424 (+ 732us) server_negotiation.cc:407] Connection header received
0806 12:24:28.531517 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:28.531521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:28.531719 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:28.531878 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:24:28.532138 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.533678 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.534834 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.535888 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.536033 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.536251 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:28.536933 (+ 682us) server_negotiation.cc:299] Negotiation successful
0806 12:24:28.537134 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":122,"threads_started":1}
I20260806 12:24:28.536825 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.529452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:33573 (local address 127.0.0.1:37972)
0806 12:24:28.531114 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:28.531128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:28.531364 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:28.531868 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:28.531871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:28.531892 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:24:28.532006 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.532012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.533804 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.533808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.534687 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.534700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.536336 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.536341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.536448 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.536479 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 12:24:28.536552 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 12:24:28.536644 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1590,"thread_start_us":136,"threads_started":1}
I20260806 12:24:28.537725 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.530928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.28.210.66:42969)
0806 12:24:28.531396 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:28.531409 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:28.531428 (+ 19us) server_negotiation.cc:407] Connection header received
0806 12:24:28.531507 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:28.531512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:28.531719 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:28.531878 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:24:28.532395 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.533481 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.535073 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.535943 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.536039 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.537350 (+ 1311us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:28.537424 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 12:24:28.537552 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":167,"threads_started":1}
I20260806 12:24:28.540700 13444 tablet_service.cc:1467] Processing CreateTablet for tablet 3be809f6c5aa411885f515a7741c19cf (DEFAULT_TABLE table=test-table [id=7eb540df6d4248e9b98136d4d54180ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:28.540958 13063 tablet_service.cc:1467] Processing CreateTablet for tablet 3be809f6c5aa411885f515a7741c19cf (DEFAULT_TABLE table=test-table [id=7eb540df6d4248e9b98136d4d54180ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:28.540984 13252 tablet_service.cc:1467] Processing CreateTablet for tablet 3be809f6c5aa411885f515a7741c19cf (DEFAULT_TABLE table=test-table [id=7eb540df6d4248e9b98136d4d54180ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:24:28.542361 13444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be809f6c5aa411885f515a7741c19cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:28.542353 13063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be809f6c5aa411885f515a7741c19cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:28.542353 13252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be809f6c5aa411885f515a7741c19cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:24:28.550454 13571 tablet_bootstrap.cc:492] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a: Bootstrap starting.
I20260806 12:24:28.551355 13572 tablet_bootstrap.cc:492] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d: Bootstrap starting.
I20260806 12:24:28.552251 13573 tablet_bootstrap.cc:492] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f: Bootstrap starting.
I20260806 12:24:28.554126 13572 tablet_bootstrap.cc:654] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:28.554216 13571 tablet_bootstrap.cc:654] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:28.554960 13573 tablet_bootstrap.cc:654] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f: Neither blocks nor log segments found. Creating new log.
I20260806 12:24:28.555037 13572 log.cc:824] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d: Log is configured to *not* fsync() on all Append() calls
I20260806 12:24:28.555619 13571 log.cc:824] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a: Log is configured to *not* fsync() on all Append() calls
I20260806 12:24:28.555815 13573 log.cc:824] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f: Log is configured to *not* fsync() on all Append() calls
I20260806 12:24:28.557741 13571 tablet_bootstrap.cc:492] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a: No bootstrap required, opened a new log
I20260806 12:24:28.557793 13572 tablet_bootstrap.cc:492] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d: No bootstrap required, opened a new log
I20260806 12:24:28.557893 13573 tablet_bootstrap.cc:492] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f: No bootstrap required, opened a new log
I20260806 12:24:28.558002 13571 ts_tablet_manager.cc:1397] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 12:24:28.558017 13572 ts_tablet_manager.cc:1397] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 12:24:28.558090 13573 ts_tablet_manager.cc:1397] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 12:24:28.562114 13571 raft_consensus.cc:348] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d93ce3a3163436085db915f54644e6a" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 46205 } } peers { permanent_uuid: "39d2a66d2ee7412c92c3399cdf60413d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33573 } } peers { permanent_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42969 } }
I20260806 12:24:28.562502 13571 raft_consensus.cc:374] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:28.562572 13571 raft_consensus.cc:729] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d93ce3a3163436085db915f54644e6a, State: Initialized, Role: FOLLOWER
I20260806 12:24:28.562443 13573 raft_consensus.cc:348] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d93ce3a3163436085db915f54644e6a" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 46205 } } peers { permanent_uuid: "39d2a66d2ee7412c92c3399cdf60413d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33573 } } peers { permanent_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42969 } }
I20260806 12:24:28.562760 13573 raft_consensus.cc:374] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:28.562840 13573 raft_consensus.cc:729] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc0a4eac7bc4b3690ff6e813d1fce0f, State: Initialized, Role: FOLLOWER
I20260806 12:24:28.563068 13572 raft_consensus.cc:348] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d93ce3a3163436085db915f54644e6a" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 46205 } } peers { permanent_uuid: "39d2a66d2ee7412c92c3399cdf60413d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33573 } } peers { permanent_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42969 } }
I20260806 12:24:28.563359 13572 raft_consensus.cc:374] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:24:28.563271 13571 consensus_queue.cc:260] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d93ce3a3163436085db915f54644e6a" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 46205 } } peers { permanent_uuid: "39d2a66d2ee7412c92c3399cdf60413d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33573 } } peers { permanent_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42969 } }
I20260806 12:24:28.563360 13573 consensus_queue.cc:260] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d93ce3a3163436085db915f54644e6a" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 46205 } } peers { permanent_uuid: "39d2a66d2ee7412c92c3399cdf60413d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33573 } } peers { permanent_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42969 } }
I20260806 12:24:28.563428 13572 raft_consensus.cc:729] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d2a66d2ee7412c92c3399cdf60413d, State: Initialized, Role: FOLLOWER
I20260806 12:24:28.564247 13572 consensus_queue.cc:260] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d93ce3a3163436085db915f54644e6a" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 46205 } } peers { permanent_uuid: "39d2a66d2ee7412c92c3399cdf60413d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33573 } } peers { permanent_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42969 } }
I20260806 12:24:28.564704 13546 heartbeater.cc:503] Master 127.28.210.126:41035 was elected leader, sending a full tablet report...
I20260806 12:24:28.564754 13573 ts_tablet_manager.cc:1428] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 12:24:28.564766 13571 ts_tablet_manager.cc:1428] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 12:24:28.565162 13345 heartbeater.cc:503] Master 127.28.210.126:41035 was elected leader, sending a full tablet report...
I20260806 12:24:28.566658 13157 heartbeater.cc:503] Master 127.28.210.126:41035 was elected leader, sending a full tablet report...
I20260806 12:24:28.567147 13572 ts_tablet_manager.cc:1428] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d: Time spent starting tablet: real 0.009s user 0.002s sys 0.006s
W20260806 12:24:28.652381 13158 tablet.cc:2369] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:24:28.652662 13581 raft_consensus.cc:482] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:24:28.652906 13581 raft_consensus.cc:504] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d93ce3a3163436085db915f54644e6a" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 46205 } } peers { permanent_uuid: "39d2a66d2ee7412c92c3399cdf60413d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33573 } } peers { permanent_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42969 } }
I20260806 12:24:28.654510 13581 leader_election.cc:288] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d93ce3a3163436085db915f54644e6a (127.28.210.67:46205), 8dc0a4eac7bc4b3690ff6e813d1fce0f (127.28.210.66:42969)
I20260806 12:24:28.659610 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.655031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:33147 (local address 127.28.210.66:42969)
0806 12:24:28.655181 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:28.655186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:28.655447 (+ 261us) server_negotiation.cc:407] Connection header received
0806 12:24:28.655580 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:28.655582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:28.655628 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:28.655705 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:24:28.656029 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.656723 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.658156 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.658870 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.658990 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.659045 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:28.659353 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 12:24:28.659451 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:24:28.659631 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.654798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:42969 (local address 127.28.210.65:33147)
0806 12:24:28.655328 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:28.655340 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:28.655471 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:28.655729 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:28.655732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:28.655749 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:24:28.655907 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.655912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.656882 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.656886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.657957 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.657974 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.659032 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.659037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.659150 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.659183 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 12:24:28.659284 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 12:24:28.659390 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":21,"thread_start_us":180,"threads_started":1}
I20260806 12:24:28.660207 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.655132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:37017 (local address 127.28.210.67:46205)
0806 12:24:28.655261 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:28.655265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:28.655404 (+ 139us) server_negotiation.cc:407] Connection header received
0806 12:24:28.655548 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:28.655552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:28.655604 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:28.655666 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:24:28.656027 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.656746 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.658820 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.659492 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.659641 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.659696 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:28.659960 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 12:24:28.660047 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 12:24:28.660661 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.654762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:46205 (local address 127.28.210.65:37017)
0806 12:24:28.655303 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:28.655315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:24:28.655434 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:24:28.655698 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:24:28.655701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:24:28.655717 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:24:28.655893 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.655900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.656909 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.656912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.658617 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:24:28.658632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.659673 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.659677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:24:28.659781 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.659807 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 12:24:28.660387 (+ 580us) client_negotiation.cc:240] Negotiation successful
0806 12:24:28.660492 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":324,"spinlock_wait_cycles":11520,"thread_start_us":222,"threads_started":1}
I20260806 12:24:28.660759 13280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be809f6c5aa411885f515a7741c19cf" candidate_uuid: "39d2a66d2ee7412c92c3399cdf60413d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f" is_pre_election: true
I20260806 12:24:28.661237 13280 raft_consensus.cc:2454] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d2a66d2ee7412c92c3399cdf60413d in term 0.
I20260806 12:24:28.661474 13475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be809f6c5aa411885f515a7741c19cf" candidate_uuid: "39d2a66d2ee7412c92c3399cdf60413d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d93ce3a3163436085db915f54644e6a" is_pre_election: true
I20260806 12:24:28.661834 13475 raft_consensus.cc:2454] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39d2a66d2ee7412c92c3399cdf60413d in term 0.
I20260806 12:24:28.661939 13002 leader_election.cc:302] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d2a66d2ee7412c92c3399cdf60413d, 8dc0a4eac7bc4b3690ff6e813d1fce0f; no voters:
I20260806 12:24:28.662274 13581 raft_consensus.cc:2781] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:24:28.662379 13581 raft_consensus.cc:482] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:24:28.662434 13581 raft_consensus.cc:3037] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:28.663901 13581 raft_consensus.cc:504] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d93ce3a3163436085db915f54644e6a" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 46205 } } peers { permanent_uuid: "39d2a66d2ee7412c92c3399cdf60413d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33573 } } peers { permanent_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42969 } }
I20260806 12:24:28.664495 13581 leader_election.cc:288] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [CANDIDATE]: Term 1 election: Requested vote from peers 2d93ce3a3163436085db915f54644e6a (127.28.210.67:46205), 8dc0a4eac7bc4b3690ff6e813d1fce0f (127.28.210.66:42969)
I20260806 12:24:28.665150 13280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be809f6c5aa411885f515a7741c19cf" candidate_uuid: "39d2a66d2ee7412c92c3399cdf60413d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f"
I20260806 12:24:28.665306 13280 raft_consensus.cc:3037] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:28.665361 13475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be809f6c5aa411885f515a7741c19cf" candidate_uuid: "39d2a66d2ee7412c92c3399cdf60413d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d93ce3a3163436085db915f54644e6a"
I20260806 12:24:28.665573 13475 raft_consensus.cc:3037] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:24:28.666620 13280 raft_consensus.cc:2454] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d2a66d2ee7412c92c3399cdf60413d in term 1.
I20260806 12:24:28.666872 13475 raft_consensus.cc:2454] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39d2a66d2ee7412c92c3399cdf60413d in term 1.
I20260806 12:24:28.667088 13002 leader_election.cc:302] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d2a66d2ee7412c92c3399cdf60413d, 8dc0a4eac7bc4b3690ff6e813d1fce0f; no voters:
I20260806 12:24:28.667366 13581 raft_consensus.cc:2781] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:24:28.667689 13581 raft_consensus.cc:686] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [term 1 LEADER]: Becoming Leader. State: Replica: 39d2a66d2ee7412c92c3399cdf60413d, State: Running, Role: LEADER
I20260806 12:24:28.668185 13581 consensus_queue.cc:237] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d93ce3a3163436085db915f54644e6a" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 46205 } } peers { permanent_uuid: "39d2a66d2ee7412c92c3399cdf60413d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33573 } } peers { permanent_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42969 } }
I20260806 12:24:28.671790 12891 catalog_manager.cc:5495] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d reported cstate change: term changed from 0 to 1, leader changed from <none> to 39d2a66d2ee7412c92c3399cdf60413d (127.28.210.65). New cstate: current_term: 1 leader_uuid: "39d2a66d2ee7412c92c3399cdf60413d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d93ce3a3163436085db915f54644e6a" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 46205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d2a66d2ee7412c92c3399cdf60413d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 33573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42969 } health_report { overall_health: UNKNOWN } } }
W20260806 12:24:28.690521 13347 tablet.cc:2369] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:24:28.695102 13567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.690315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.28.210.65:33573)
0806 12:24:28.690472 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 12:24:28.690478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:24:28.690523 (+ 45us) server_negotiation.cc:407] Connection header received
0806 12:24:28.690610 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:24:28.690612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:24:28.690689 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:24:28.690790 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:24:28.691117 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.692036 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.693284 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:24:28.693538 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:24:28.693648 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:24:28.694053 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:24:28.694643 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:24:28.694751 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 12:24:28.694849 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 12:24:28.694919 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 12:24:28.703172 13475 raft_consensus.cc:1261] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a [term 1 FOLLOWER]: Refusing update from remote peer 39d2a66d2ee7412c92c3399cdf60413d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:28.703172 13280 raft_consensus.cc:1261] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f [term 1 FOLLOWER]: Refusing update from remote peer 39d2a66d2ee7412c92c3399cdf60413d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:24:28.703958 13581 consensus_queue.cc:1035] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d93ce3a3163436085db915f54644e6a" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 46205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:28.704414 13584 consensus_queue.cc:1035] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc0a4eac7bc4b3690ff6e813d1fce0f" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:24:28.724152 12889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:24:28.724403 12889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:24:28.728443 12889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cad373ae4e14544abff7a2b2a7fa4f3: Sending DeleteTablet for 3 replicas of tablet 3be809f6c5aa411885f515a7741c19cf
I20260806 12:24:28.729513 13444 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be809f6c5aa411885f515a7741c19cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:28 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:44732
I20260806 12:24:28.729576 13252 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be809f6c5aa411885f515a7741c19cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:28 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:56730
I20260806 12:24:28.729669 13063 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be809f6c5aa411885f515a7741c19cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:24:28 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:37972
I20260806 12:24:28.730212 13604 tablet_replica.cc:330] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f: stopping tablet replica
I20260806 12:24:28.730480 13600 tablet_replica.cc:330] T 3be809f6c5aa411885f515a7741c19cf P 39d2a66d2ee7412c92c3399cdf60413d: stopping tablet replica
I20260806 12:24:28.730643 13604 raft_consensus.cc:2229] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:24:28.730690 13602 tablet_replica.cc:330] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a: stopping tablet replica
I20260806 12:24:28.731035 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 12976
I20260806 12:24:28.731086 13604 raft_consensus.cc:2258] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:28.735786 13604 ts_tablet_manager.cc:1905] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:28.738785 13604 ts_tablet_manager.cc:1918] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:24:28.738904 13604 log.cc:1196] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/3be809f6c5aa411885f515a7741c19cf
I20260806 12:24:28.739197 13604 ts_tablet_manager.cc:1939] T 3be809f6c5aa411885f515a7741c19cf P 8dc0a4eac7bc4b3690ff6e813d1fce0f: Deleting consensus metadata
I20260806 12:24:28.740772 13602 raft_consensus.cc:2229] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:24:28.741180 13602 raft_consensus.cc:2258] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:24:28.741820 12871 catalog_manager.cc:4841] TS 8dc0a4eac7bc4b3690ff6e813d1fce0f (127.28.210.66:42969): tablet 3be809f6c5aa411885f515a7741c19cf (table test-table [id=7eb540df6d4248e9b98136d4d54180ed]) successfully deleted
I20260806 12:24:28.742902 13602 ts_tablet_manager.cc:1905] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:24:28.745507 13602 ts_tablet_manager.cc:1918] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:24:28.745618 13602 log.cc:1196] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/3be809f6c5aa411885f515a7741c19cf
I20260806 12:24:28.745900 13602 ts_tablet_manager.cc:1939] T 3be809f6c5aa411885f515a7741c19cf P 2d93ce3a3163436085db915f54644e6a: Deleting consensus metadata
I20260806 12:24:28.747434 12872 catalog_manager.cc:4841] TS 2d93ce3a3163436085db915f54644e6a (127.28.210.67:46205): tablet 3be809f6c5aa411885f515a7741c19cf (table test-table [id=7eb540df6d4248e9b98136d4d54180ed]) successfully deleted
W20260806 12:24:28.748147 12870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:33573 (error 108)
I20260806 12:24:28.749082 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 13165
I20260806 12:24:28.749356 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:24:28.749030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:33573 (local address 127.0.0.1:37988)
0806 12:24:28.749181 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 12:24:28.749258 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 12:24:28.760517 12870 catalog_manager.cc:4584] TS 39d2a66d2ee7412c92c3399cdf60413d (127.28.210.65:33573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3be809f6c5aa411885f515a7741c19cf: Network error: Client connection negotiation failed: client connection to 127.28.210.65:33573: connect: Connection refused (error 111)
I20260806 12:24:28.763700 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 13355
I20260806 12:24:28.774487 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 12853
2026-08-06T12:24:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2792 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:24:28 dist-test-slave-lgr5 krb5kdc[13610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:24:28 dist-test-slave-lgr5 krb5kdc[13610](info): set up 2 sockets
Aug 06 12:24:28 dist-test-slave-lgr5 krb5kdc[13610](info): commencing operation
krb5kdc: starting...
W20260806 12:24:29.953498 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:24:30 dist-test-slave-lgr5 krb5kdc[13610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:24:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/rangeradmin_127.28.210.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/rangeradmin_127.28.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/rangerlookup_127.28.210.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/rangerlookup_127.28.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/HTTP_127.28.210.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/HTTP_127.28.210.84@KRBTEST.COM.keytab.
I20260806 12:24:30.164204 29513 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPswvRx/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/postgres -l logfile start
2026-08-06 12:24:33.084 UTC [13655] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 12:24:33.085 UTC [13655] LOG: listening on IPv4 address "127.28.210.84", port 56913
2026-08-06 12:24:33.089 UTC [13655] LOG: listening on Unix socket "/tmp/.s.PGSQL.56913"
2026-08-06 12:24:33.115 UTC [13660] LOG: database system was shut down at 2026-08-06 12:24:31 UTC
2026-08-06 12:24:33.119 UTC [13655] LOG: database system is ready to accept connections
I20260806 12:24:34.210513 29513 mini_postgres.cc:96] Postgres bound to 56913
2026-08-06 12:24:34.220 UTC [13669] FATAL: database "slave" does not exist
127.28.210.84:56913 - accepting connections
I20260806 12:24:34.221276 29513 mini_ranger.cc:161] Starting Ranger...
I20260806 12:24:34.242033 29513 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 12:24:34.822293 29513 mini_ranger.cc:87] Created ranger Postgres database
I20260806 12:24:34.822400 29513 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-admin
2026-08-06 12:24:35,171 [I] DB FLAVOR :POSTGRES
2026-08-06 12:24:35,172 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 12:24:35,172 [I] Checking connection..
2026-08-06 12:24:35,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 12:24:35,464 [I] Checking connection passed.
2026-08-06 12:24:35,464 [I] --------- Verifying version history table ---------
2026-08-06 12:24:35,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 12:24:35,750 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 12:24:35,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 12:24:36,043 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 12:24:36,044 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 12:24:36,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 12:24:36.319 UTC [13763] WARNING: there is no transaction in progress
2026-08-06 12:24:36,329 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 12:24:36,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 12:24:36,629 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 12:24:36,629 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 12:24:36,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 12:24:36,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 12:24:37,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lgr5.c.gcp-upstream.internal','N') ;"
2026-08-06 12:24:37,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 12:24:37,738 [I] Table x_portal_user does not exist in database ranger
2026-08-06 12:24:37,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 12:24:38,039 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 12:24:38,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 12:24:38,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 12:24:38,595 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 12:24:38,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 12:24:39.790 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:39.811 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:39.833 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:39.853 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:39.873 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:39.895 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:39.961 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:39.970 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:39.989 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:40.004 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:40.018 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:40.035 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:40.050 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:40.059 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:40.069 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:40.079 UTC [13918] WARNING: there is no transaction in progress
2026-08-06 12:24:41,037 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 12:24:41,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 12:24:41,334 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 12:24:41,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 12:24:41,640 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 12:24:41,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 12:24:41,937 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 12:24:41,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 12:24:42,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 12:24:42,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lgr5.c.gcp-upstream.internal';"
2026-08-06 12:24:42,799 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 12:24:42,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 12:24:43,107 [I] DB_PATCHES have already been applied
I20260806 12:24:43.113927 29513 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPswvRx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 12:24:43.114015 29513 mini_ranger.cc:193] Using host: 127.28.210.84
I20260806 12:24:43.119287 29513 mini_ranger.cc:227] Ranger admin URL: http://127.28.210.84:35515
Aug 06, 2026 12:24:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 12:24:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 12:24:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 12:24:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 12:24:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 12:24:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 12:24:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 12:24:44.398231 29513 mini_ranger.cc:160] Time spent starting Ranger: real 10.177s user 0.000s sys 0.014s
Aug 06, 2026 12:25:08 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 12:25:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 12:25:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 12:25:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 12:25:08 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 12:25:10 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 12:25:10.853099 29513 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/rangerkms_127.28.210.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/rangerkms_127.28.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.210.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.210.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.210.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/HTTP_127.28.210.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.210.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/HTTP_127.28.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 12:25:10 dist-test-slave-lgr5 krb5kdc[13610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019110, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 12:25:10.970988 29513 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 12:25:10.989706 29513 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 12:25:12.245250 29513 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 12:25:12.245355 29513 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-kms
I20260806 12:25:12.245469 29513 mini_ranger_kms.cc:227] Using postgres at 127.28.210.84:56913
2026-08-06 12:25:13,740 [I] DB FLAVOR :POSTGRES
2026-08-06 12:25:13,740 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 12:25:13,740 [I] Checking connection
2026-08-06 12:25:13,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 12:25:14,143 [I] connection success
2026-08-06 12:25:14,144 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 12:25:14,144 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 12:25:14,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 12:25:14,427 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 12:25:14,427 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 12:25:14,428 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 12:25:14,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:56913/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 12:25:14,745 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 12:25:15.404820 29513 mini_ranger_kms.cc:310] Created kms service
I20260806 12:25:15.673647 29513 mini_ranger_kms.cc:326] Created kudu user
I20260806 12:25:15.728868 29513 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 12:25:15 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019110, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:25:16.224673 29513 mini_ranger_kms.cc:384] Added ranger policy
I20260806 12:25:16.224867 29513 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 12:25:16.224891 29513 mini_ranger_kms.cc:240] Using host: 127.28.210.84
I20260806 12:25:16.230291 29513 mini_ranger_kms.cc:276] Ranger KMS PID: 14180
I20260806 12:25:16.230537 29513 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.210.84:42465
Aug 06, 2026 12:25:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 12:25:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 12:25:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 12:25:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 12:25:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 12:25:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 12:25:18.884497 29513 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 7.914s user 0.000s sys 1.113s
I20260806 12:25:18.884680 29513 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 12:25:18.884810 29513 mini_ranger_kms.cc:401] 127.28.210.84:42465/kms/v1/keys
Aug 06 12:25:19 dist-test-slave-lgr5 krb5kdc[13610](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786019119, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.210.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 12:25:19 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019119, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.210.84@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
Aug 06 12:25:20 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019119, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.210.84@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
Aug 06 12:25:23 dist-test-slave-lgr5 krb5kdc[13610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:25:23.093371 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:42373
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:42373
--ranger_config_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.210.84:42465/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:25:23.255177 14257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:25:23.255575 14257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:25:23.255640 14257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:25:23.260584 14257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:25:23.260736 14257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:25:23.260766 14257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:25:23.260788 14257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:25:23.260810 14257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:25:23.268054 14257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.210.84:42465/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:42373
--ranger_config_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:42373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:25:23.269582 14257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:25:23.270915 14257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:25:23.283778 14263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:25:23.283747 14266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:25:23.283876 14257 server_base.cc:1034] running on GCE node
W20260806 12:25:23.283747 14268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:25:23.285202 14257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:25:23.286702 14257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:25:23.287937 14257 hybrid_clock.cc:648] HybridClock initialized: now 1786019123287927 us; error 43 us; skew 500 ppm
Aug 06 12:25:23 dist-test-slave-lgr5 krb5kdc[13610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019123, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:25:23.295871 14257 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:25:23.297325 14257 webserver.cc:460] Webserver started at http://127.28.210.126:45597/ using document root <none> and password file <none>
I20260806 12:25:23.298022 14257 fs_manager.cc:359] Metadata directory not provided
I20260806 12:25:23.298097 14257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:25:23.298341 14257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 12:25:23 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019123, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:25:23.371469 14257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "0ca14e98415748728421370d462eee86"
format_stamp: "Formatted at 2026-08-06 12:25:23 on dist-test-slave-lgr5"
server_key: "a508d883429d30c07f90427b0f747470"
server_key_iv: "3d608055bcb38d3952cd172c481b070b"
server_key_version: "kuduclusterkey@0"
I20260806 12:25:23.372406 14257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "0ca14e98415748728421370d462eee86"
format_stamp: "Formatted at 2026-08-06 12:25:23 on dist-test-slave-lgr5"
server_key: "a508d883429d30c07f90427b0f747470"
server_key_iv: "3d608055bcb38d3952cd172c481b070b"
server_key_version: "kuduclusterkey@0"
I20260806 12:25:23.379453 14257 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260806 12:25:23.401011 14284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:25:23.403012 14257 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:25:23.403208 14257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "0ca14e98415748728421370d462eee86"
format_stamp: "Formatted at 2026-08-06 12:25:23 on dist-test-slave-lgr5"
server_key: "a508d883429d30c07f90427b0f747470"
server_key_iv: "3d608055bcb38d3952cd172c481b070b"
server_key_version: "kuduclusterkey@0"
I20260806 12:25:23.403384 14257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:25:23.443692 14257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:25:23.447115 14257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:25:23.447336 14257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:25:23.460443 14257 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:42373
I20260806 12:25:23.460554 14358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:42373 every 8 connection(s)
I20260806 12:25:23.461581 14257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:25:23.465044 14360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:25:23.465876 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 14257
I20260806 12:25:23.465998 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:25:23.466270 29513 external_mini_cluster.cc:1428] Setting key 8f22f2a968b71aea55ba6851255e5e5a
I20260806 12:25:23.473819 14360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86: Bootstrap starting.
I20260806 12:25:23.477550 14360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86: Neither blocks nor log segments found. Creating new log.
Aug 06 12:25:23 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:25:23.478899 14360 log.cc:824] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86: Log is configured to *not* fsync() on all Append() calls
I20260806 12:25:23.482133 14360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86: No bootstrap required, opened a new log
I20260806 12:25:23.486387 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:23.467748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.28.210.126:42373)
0806 12:25:23.468502 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0806 12:25:23.468565 (+ 63us) server_negotiation.cc:399] Waiting for connection header
0806 12:25:23.468616 (+ 51us) server_negotiation.cc:407] Connection header received
0806 12:25:23.470064 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:25:23.470092 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:25:23.470551 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:25:23.470961 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:25:23.471739 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:23.473565 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:23.474685 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:23.475557 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:23.475777 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:23.479075 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:25:23.479083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:25:23.479133 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:25:23.479179 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:25:23.482589 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:25:23.483130 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:25:23.483137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:25:23.483144 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:25:23.483296 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:25:23.483674 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:25:23.483678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:25:23.483680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:25:23.484339 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:25:23.484548 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 12:25:23.484871 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 12:25:23.485217 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":273,"threads_started":1}
I20260806 12:25:23.487568 14360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ca14e98415748728421370d462eee86" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42373 } }
I20260806 12:25:23.487876 14360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:25:23.487949 14360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ca14e98415748728421370d462eee86, State: Initialized, Role: FOLLOWER
I20260806 12:25:23.488482 14360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ca14e98415748728421370d462eee86" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42373 } }
I20260806 12:25:23.488615 14360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:25:23.488674 14360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:25:23.488773 14360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:25:23.489914 14360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ca14e98415748728421370d462eee86" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42373 } }
I20260806 12:25:23.490275 14360 leader_election.cc:302] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ca14e98415748728421370d462eee86; no voters:
I20260806 12:25:23.490630 14360 leader_election.cc:288] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:25:23.490911 14365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:25:23.491235 14365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [term 1 LEADER]: Becoming Leader. State: Replica: 0ca14e98415748728421370d462eee86, State: Running, Role: LEADER
I20260806 12:25:23.491580 14365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ca14e98415748728421370d462eee86" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42373 } }
I20260806 12:25:23.492235 14360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:25:23.493860 14366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ca14e98415748728421370d462eee86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ca14e98415748728421370d462eee86" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42373 } } }
I20260806 12:25:23.494017 14366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [sys.catalog]: This master's current role is: LEADER
I20260806 12:25:23.493853 14367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ca14e98415748728421370d462eee86. Latest consensus state: current_term: 1 leader_uuid: "0ca14e98415748728421370d462eee86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ca14e98415748728421370d462eee86" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42373 } } }
I20260806 12:25:23.494382 14367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86 [sys.catalog]: This master's current role is: LEADER
I20260806 12:25:23.494719 14380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:25:23.498035 14380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:25:23.498924 14360 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 12:25:23.508716 14380 catalog_manager.cc:1347] Generated new cluster ID: 80e5c98b3731455a82a095be068a454d
I20260806 12:25:23.508813 14380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:25:23.519284 14380 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:25:23.520591 14380 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:25:23.532001 14380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86: Generated new TSK 0
I20260806 12:25:23.532900 14380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:25:24.531703 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.210.84:42465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42373
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:25:24.732982 14425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:25:24.733335 14425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:25:24.733402 14425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:25:24.738164 14425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:25:24.738265 14425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:25:24.738356 14425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:25:24.744913 14425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.210.84:42465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:25:24.746675 14425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:25:24.748297 14425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:25:24.765810 14434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:25:24.766152 14433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:25:24.766477 14436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:25:24.766458 14425 server_base.cc:1034] running on GCE node
I20260806 12:25:24.767153 14425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:25:24.767939 14425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:25:24.769121 14425 hybrid_clock.cc:648] HybridClock initialized: now 1786019124769094 us; error 46 us; skew 500 ppm
Aug 06 12:25:24 dist-test-slave-lgr5 krb5kdc[13610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019124, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:25:24.772727 14425 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:25:24.773810 14425 webserver.cc:460] Webserver started at http://127.28.210.65:41671/ using document root <none> and password file <none>
I20260806 12:25:24.774420 14425 fs_manager.cc:359] Metadata directory not provided
I20260806 12:25:24.774485 14425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:25:24.774731 14425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 12:25:24 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019124, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:25:24.800431 14425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "1be586c87a964245b8433f44fe10ce43"
format_stamp: "Formatted at 2026-08-06 12:25:24 on dist-test-slave-lgr5"
server_key: "5744e18715e200d441e587c67c928175"
server_key_iv: "5b82396ea40b067cb88df14381e90894"
server_key_version: "kuduclusterkey@0"
I20260806 12:25:24.801674 14425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "1be586c87a964245b8433f44fe10ce43"
format_stamp: "Formatted at 2026-08-06 12:25:24 on dist-test-slave-lgr5"
server_key: "5744e18715e200d441e587c67c928175"
server_key_iv: "5b82396ea40b067cb88df14381e90894"
server_key_version: "kuduclusterkey@0"
I20260806 12:25:24.807832 14425 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260806 12:25:24.828684 14453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:25:24.830951 14425 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260806 12:25:24.831154 14425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "1be586c87a964245b8433f44fe10ce43"
format_stamp: "Formatted at 2026-08-06 12:25:24 on dist-test-slave-lgr5"
server_key: "5744e18715e200d441e587c67c928175"
server_key_iv: "5b82396ea40b067cb88df14381e90894"
server_key_version: "kuduclusterkey@0"
I20260806 12:25:24.831326 14425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:25:24.862953 14425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:25:24.866536 14425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:25:24.866748 14425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:25:24.867476 14425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:25:24.868737 14425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:25:24.868848 14425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 12:25:24.868933 14425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:25:24.868973 14425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:25:24.885116 14425 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:43123
I20260806 12:25:24.886597 14425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:25:24.889484 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 14425
I20260806 12:25:24.889618 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:25:24.889950 29513 external_mini_cluster.cc:1428] Setting key 7d6ecbad3fc82afe6bcfadec56b8ab5f
I20260806 12:25:24.897836 14621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:43123 every 8 connection(s)
Aug 06 12:25:24 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019124, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:25:24.917872 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:24.901439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:58273 (local address 127.28.210.126:42373)
0806 12:25:24.904247 (+ 2808us) server_negotiation.cc:206] Beginning negotiation
0806 12:25:24.904252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:25:24.904269 (+ 17us) server_negotiation.cc:407] Connection header received
0806 12:25:24.904322 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:25:24.904324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:25:24.904387 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:25:24.904494 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:25:24.906260 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:24.907105 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:24.909068 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:24.909413 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:24.909546 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:24.913336 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:25:24.913342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:25:24.913345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:25:24.913383 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:25:24.915335 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:25:24.916107 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:25:24.916112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:25:24.916114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:25:24.916215 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:25:24.916592 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:25:24.916595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:25:24.916597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:25:24.916858 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:25:24.916962 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 12:25:24.917538 (+ 576us) server_negotiation.cc:299] Negotiation successful
0806 12:25:24.917680 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2652,"thread_start_us":76,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:25:24.919085 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:24.900061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:42373 (local address 127.28.210.65:58273)
0806 12:25:24.902389 (+ 2328us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:24.902456 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 12:25:24.903680 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:25:24.904860 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:25:24.904871 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:25:24.905272 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:25:24.906051 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:24.906072 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:24.907257 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:24.907261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:24.908876 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:24.908893 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:24.909835 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:24.909842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:24.910138 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:24.910918 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:25:24.910943 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:25:24.913048 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:25:24.915504 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:25:24.915512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:25:24.915534 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:25:24.915952 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:25:24.916307 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:25:24.916311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:25:24.916313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:25:24.916479 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:25:24.916963 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:25:24.916971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:25:24.917274 (+ 303us) client_negotiation.cc:769] Sending connection context
0806 12:25:24.917459 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 12:25:24.917834 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2031,"thread_start_us":186,"threads_started":1}
I20260806 12:25:24.921722 14622 heartbeater.cc:347] Connected to a master server at 127.28.210.126:42373
I20260806 12:25:24.922047 14622 heartbeater.cc:464] Registering TS with master...
I20260806 12:25:24.922689 14622 heartbeater.cc:511] Master 127.28.210.126:42373 requested a full tablet report, sending...
I20260806 12:25:24.924621 14308 ts_manager.cc:194] Registered new tserver with Master: 1be586c87a964245b8433f44fe10ce43 (127.28.210.65:43123)
I20260806 12:25:24.926210 14308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:58273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:25:24.944164 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.210.84:42465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42373
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 12:25:24 dist-test-slave-lgr5 krb5kdc[13610](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786019124, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 12:25:25.173806 14634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:25:25.174358 14634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:25:25.174469 14634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:25:25.180027 14634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:25:25.180194 14634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:25:25.180291 14634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:25:25.188239 14634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.210.84:42465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:25:25.190368 14634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:25:25.192322 14634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:25:25.212198 14642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:25:25.213534 14645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:25:25.213909 14641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:25:25.214609 14634 server_base.cc:1034] running on GCE node
I20260806 12:25:25.215597 14634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:25:25.216960 14634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:25:25.218183 14634 hybrid_clock.cc:648] HybridClock initialized: now 1786019125218151 us; error 64 us; skew 500 ppm
Aug 06 12:25:25 dist-test-slave-lgr5 krb5kdc[13610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019125, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:25:25.224118 14634 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:25:25.225605 14634 webserver.cc:460] Webserver started at http://127.28.210.66:45375/ using document root <none> and password file <none>
I20260806 12:25:25.226517 14634 fs_manager.cc:359] Metadata directory not provided
I20260806 12:25:25.226697 14634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:25:25.227052 14634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 12:25:25 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019125, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:25:25.248826 14634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7"
format_stamp: "Formatted at 2026-08-06 12:25:25 on dist-test-slave-lgr5"
server_key: "3b62737817ee76a81c993455c20626ad"
server_key_iv: "f4af450d0d6f3d030e62e1892ac7d281"
server_key_version: "kuduclusterkey@0"
I20260806 12:25:25.249918 14634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7"
format_stamp: "Formatted at 2026-08-06 12:25:25 on dist-test-slave-lgr5"
server_key: "3b62737817ee76a81c993455c20626ad"
server_key_iv: "f4af450d0d6f3d030e62e1892ac7d281"
server_key_version: "kuduclusterkey@0"
I20260806 12:25:25.257934 14634 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260806 12:25:25.284766 14653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:25:25.286891 14634 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.000s
I20260806 12:25:25.287211 14634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7"
format_stamp: "Formatted at 2026-08-06 12:25:25 on dist-test-slave-lgr5"
server_key: "3b62737817ee76a81c993455c20626ad"
server_key_iv: "f4af450d0d6f3d030e62e1892ac7d281"
server_key_version: "kuduclusterkey@0"
I20260806 12:25:25.288057 14634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:25:25.307794 14634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:25:25.313130 14634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:25:25.313459 14634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:25:25.314409 14634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:25:25.315882 14634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:25:25.316006 14634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 12:25:25.316140 14634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:25:25.316205 14634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:25:25.346848 14634 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:43947
I20260806 12:25:25.348101 14634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:25:25.353262 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 14634
I20260806 12:25:25.353442 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:25:25.353592 14826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:43947 every 8 connection(s)
I20260806 12:25:25.353772 29513 external_mini_cluster.cc:1428] Setting key 114859523dc45c8236b31e7fe82c0c87
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
Aug 06 12:25:25 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019125, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:25:25.385996 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:25.360571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:33405 (local address 127.28.210.126:42373)
0806 12:25:25.360751 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 12:25:25.360756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:25:25.363702 (+ 2946us) server_negotiation.cc:407] Connection header received
0806 12:25:25.365655 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:25:25.365659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:25:25.365739 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:25:25.365803 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:25:25.368900 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.369802 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.372210 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.372591 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.372697 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:25.379191 (+ 6494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:25:25.379197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:25:25.379200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:25:25.379244 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:25:25.381499 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:25:25.383903 (+ 2404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:25:25.383907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:25:25.383908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:25:25.384034 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:25:25.384493 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:25:25.384496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:25:25.384497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:25:25.384720 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:25:25.384818 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 12:25:25.385597 (+ 779us) server_negotiation.cc:299] Negotiation successful
0806 12:25:25.385785 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 12:25:25.388492 14834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:25.360283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:42373 (local address 127.28.210.66:33405)
0806 12:25:25.363424 (+ 3141us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:25.363506 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0806 12:25:25.365271 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:25:25.366416 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:25:25.366430 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:25:25.367075 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:25:25.368590 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:25.368627 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.370092 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.370096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:25.371922 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:25.371945 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.373025 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.373030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:25.373286 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:25.374189 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:25:25.374209 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:25:25.378802 (+ 4593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:25:25.383182 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:25:25.383194 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:25:25.383223 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:25:25.383723 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:25:25.384161 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:25:25.384164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:25:25.384166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:25:25.384367 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:25:25.384837 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:25:25.384846 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:25:25.385251 (+ 405us) client_negotiation.cc:769] Sending connection context
0806 12:25:25.386634 (+ 1383us) client_negotiation.cc:240] Negotiation successful
0806 12:25:25.387083 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2804,"thread_start_us":330,"threads_started":1}
I20260806 12:25:25.390666 14830 heartbeater.cc:347] Connected to a master server at 127.28.210.126:42373
I20260806 12:25:25.391000 14830 heartbeater.cc:464] Registering TS with master...
I20260806 12:25:25.391739 14830 heartbeater.cc:511] Master 127.28.210.126:42373 requested a full tablet report, sending...
I20260806 12:25:25.393343 14308 ts_manager.cc:194] Registered new tserver with Master: 78d4bcf663ab40a0bf84bc9bd7b10fe7 (127.28.210.66:43947)
I20260806 12:25:25.394248 14308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:33405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:25:25.407421 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.210.84:42465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42373
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:25:25.655457 14837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:25:25.656426 14837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:25:25.656658 14837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:25:25.664155 14837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:25:25.664276 14837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:25:25.664386 14837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
Aug 06 12:25:25 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019124, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:25:25.673836 14837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.210.84:42465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:25:25.676221 14837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:25:25.678121 14837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:25:25.698436 14846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:25:25.698940 14845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:25:25.699923 14837 server_base.cc:1034] running on GCE node
W20260806 12:25:25.700344 14849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:25:25.700954 14837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:25:25.701901 14837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:25:25.703150 14837 hybrid_clock.cc:648] HybridClock initialized: now 1786019125703074 us; error 100 us; skew 500 ppm
Aug 06 12:25:25 dist-test-slave-lgr5 krb5kdc[13610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019125, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:25:25.708536 14837 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:25:25.710402 14837 webserver.cc:460] Webserver started at http://127.28.210.67:35629/ using document root <none> and password file <none>
I20260806 12:25:25.711542 14837 fs_manager.cc:359] Metadata directory not provided
I20260806 12:25:25.711747 14837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:25:25.712186 14837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 12:25:25 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019125, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:25:25.746148 14837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "28b882660c37408e97e4834424ad68fa"
format_stamp: "Formatted at 2026-08-06 12:25:25 on dist-test-slave-lgr5"
server_key: "649187b2b6a05bc3c8a4a55ec84ebfe1"
server_key_iv: "24b70ba16c74dcc6c7b47467f64850a0"
server_key_version: "kuduclusterkey@0"
I20260806 12:25:25.747537 14837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "28b882660c37408e97e4834424ad68fa"
format_stamp: "Formatted at 2026-08-06 12:25:25 on dist-test-slave-lgr5"
server_key: "649187b2b6a05bc3c8a4a55ec84ebfe1"
server_key_iv: "24b70ba16c74dcc6c7b47467f64850a0"
server_key_version: "kuduclusterkey@0"
I20260806 12:25:25.764074 14837 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.005s sys 0.000s
I20260806 12:25:25.784554 14862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:25:25.786563 14837 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 12:25:25.786720 14837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "28b882660c37408e97e4834424ad68fa"
format_stamp: "Formatted at 2026-08-06 12:25:25 on dist-test-slave-lgr5"
server_key: "649187b2b6a05bc3c8a4a55ec84ebfe1"
server_key_iv: "24b70ba16c74dcc6c7b47467f64850a0"
server_key_version: "kuduclusterkey@0"
I20260806 12:25:25.786870 14837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:25:25.827384 14837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:25:25.832646 14837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:25:25.832999 14837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:25:25.834023 14837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:25:25.835289 14837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:25:25.835354 14837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:25:25.835409 14837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:25:25.835490 14837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 12:25:25.870034 14837 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:36713
I20260806 12:25:25.871513 15038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:36713 every 8 connection(s)
I20260806 12:25:25.872666 14837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:25:25.878808 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 14837
I20260806 12:25:25.878968 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:25:25.879276 29513 external_mini_cluster.cc:1428] Setting key 4ebbad989c8a71e9e28e8f74e26495cb
Aug 06 12:25:25 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019125, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:25:25.894084 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:25.878384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:54773 (local address 127.28.210.126:42373)
0806 12:25:25.879722 (+ 1338us) server_negotiation.cc:206] Beginning negotiation
0806 12:25:25.879727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:25:25.879749 (+ 22us) server_negotiation.cc:407] Connection header received
0806 12:25:25.879799 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:25:25.879809 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:25:25.879865 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:25:25.879968 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:25:25.881839 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.882627 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.884725 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.885095 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.885178 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:25.889252 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:25:25.889257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:25:25.889260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:25:25.889296 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:25:25.891108 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:25:25.891900 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:25:25.891904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:25:25.891906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:25:25.892010 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:25:25.892540 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:25:25.892543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:25:25.892545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:25:25.892808 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:25:25.892892 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 12:25:25.893769 (+ 877us) server_negotiation.cc:299] Negotiation successful
0806 12:25:25.893913 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225}
I20260806 12:25:25.896154 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:25.876832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:42373 (local address 127.28.210.67:54773)
0806 12:25:25.877910 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:25.877982 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 12:25:25.879432 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:25:25.880260 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:25:25.880273 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:25:25.880727 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:25:25.881654 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:25.881680 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.882807 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.882810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:25.884554 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:25.884570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.885432 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.885448 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:25.885761 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:25.886619 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:25:25.886644 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:25:25.888980 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:25:25.891283 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:25:25.891290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:25:25.891312 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:25:25.891758 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:25:25.892172 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:25:25.892175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:25:25.892177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:25:25.892401 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:25:25.892964 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:25:25.892971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:25:25.893463 (+ 492us) client_negotiation.cc:769] Sending connection context
0806 12:25:25.894288 (+ 825us) client_negotiation.cc:240] Negotiation successful
0806 12:25:25.894825 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":356,"threads_started":1}
I20260806 12:25:25.897873 15039 heartbeater.cc:347] Connected to a master server at 127.28.210.126:42373
I20260806 12:25:25.898146 15039 heartbeater.cc:464] Registering TS with master...
I20260806 12:25:25.898833 15039 heartbeater.cc:511] Master 127.28.210.126:42373 requested a full tablet report, sending...
I20260806 12:25:25.900202 14308 ts_manager.cc:194] Registered new tserver with Master: 28b882660c37408e97e4834424ad68fa (127.28.210.67:36713)
I20260806 12:25:25.901013 14308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:54773
I20260806 12:25:25.906427 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:25:25.929282 14622 heartbeater.cc:503] Master 127.28.210.126:42373 was elected leader, sending a full tablet report...
I20260806 12:25:25.932727 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:25.912690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.28.210.126:42373)
0806 12:25:25.912844 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 12:25:25.912936 (+ 92us) server_negotiation.cc:399] Waiting for connection header
0806 12:25:25.919608 (+ 6672us) server_negotiation.cc:407] Connection header received
0806 12:25:25.920574 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:25:25.920583 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:25:25.920663 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:25:25.920729 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:25:25.922142 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.923055 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.924513 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.924891 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.925171 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:25.927648 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:25:25.927654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:25:25.927658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:25:25.927711 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:25:25.930144 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:25:25.930906 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:25:25.930912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:25:25.930914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:25:25.931029 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:25:25.931412 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:25:25.931416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:25:25.931418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:25:25.931724 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:25:25.931858 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 12:25:25.932352 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 12:25:25.932526 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:25:25.938424 14308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:25:25.943598 14308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:25:25.990998 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:25.959567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:43947 (local address 127.0.0.1:38026)
0806 12:25:25.966135 (+ 6568us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:25.966214 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0806 12:25:25.966482 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:25:25.969152 (+ 2670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:25:25.969159 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:25:25.969184 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:25:25.969322 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:25.969327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.970782 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.970785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:25.985462 (+ 14677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:25.985503 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.990276 (+ 4773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.990280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:25.990441 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:25.990517 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 12:25:25.990651 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 12:25:25.990817 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6283,"thread_start_us":309,"threads_started":1}
I20260806 12:25:25.991714 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:25.959719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.28.210.66:43947)
0806 12:25:25.968330 (+ 8611us) server_negotiation.cc:206] Beginning negotiation
0806 12:25:25.968340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 12:25:25.968369 (+ 29us) server_negotiation.cc:407] Connection header received
0806 12:25:25.968463 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:25:25.968471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:25:25.968778 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:25:25.969039 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:25:25.969455 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.970632 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.985795 (+ 15163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.986792 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.986909 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:25.987098 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 12:25:25.991399 (+ 4301us) server_negotiation.cc:299] Negotiation successful
0806 12:25:25.991558 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8472,"thread_start_us":192,"threads_started":1}
I20260806 12:25:25.998379 14727 tablet_service.cc:1467] Processing CreateTablet for tablet 5c74645225f04889b49211d7deba3836 (DEFAULT_TABLE table=test-table [id=912b813ba7614bf2a9ec9f8a75ecd245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:25:25.999914 14727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c74645225f04889b49211d7deba3836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:25:26.012790 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:25.962720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:43123 (local address 127.0.0.1:53804)
0806 12:25:25.966832 (+ 4112us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:25.966846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:25:25.967166 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:25:25.974959 (+ 7793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:25:25.974963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:25:25.974985 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:25:25.976412 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:25.976426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.992331 (+ 15905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.992335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:25.993839 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:25.993854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.012275 (+ 18421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.012279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:26.012413 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:26.012448 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 12:25:26.012507 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 12:25:26.012604 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4040,"thread_start_us":267,"threads_started":1}
I20260806 12:25:26.014204 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:25.962897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.28.210.65:43123)
0806 12:25:25.974123 (+ 11226us) server_negotiation.cc:206] Beginning negotiation
0806 12:25:25.974130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:25:25.974158 (+ 28us) server_negotiation.cc:407] Connection header received
0806 12:25:25.974243 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:25:25.974249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:25:25.974485 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:25:25.974682 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:25:25.980271 (+ 5589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.981381 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:25.994083 (+ 12702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.009185 (+ 15102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.013442 (+ 4257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:26.013810 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0806 12:25:26.013895 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 12:25:26.014038 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11088,"thread_start_us":321,"threads_started":1}
I20260806 12:25:26.015035 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:25.963788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:36713 (local address 127.0.0.1:43572)
0806 12:25:25.968509 (+ 4721us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:25.968526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:25:25.968842 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:25:25.973575 (+ 4733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:25:25.973579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:25:25.973600 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:25:25.973779 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:25.973788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.009413 (+ 35625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.009418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:26.010601 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:26.010621 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.014553 (+ 3932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.014558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:26.014682 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:26.014712 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 12:25:26.014768 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 12:25:26.014864 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3574,"thread_start_us":184,"threads_started":1}
I20260806 12:25:26.019672 14530 tablet_service.cc:1467] Processing CreateTablet for tablet 5c74645225f04889b49211d7deba3836 (DEFAULT_TABLE table=test-table [id=912b813ba7614bf2a9ec9f8a75ecd245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:25:26.021210 14530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c74645225f04889b49211d7deba3836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:25:26.026794 15061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:25.964388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.28.210.67:36713)
0806 12:25:25.972389 (+ 8001us) server_negotiation.cc:206] Beginning negotiation
0806 12:25:25.972448 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0806 12:25:25.972578 (+ 130us) server_negotiation.cc:407] Connection header received
0806 12:25:25.972697 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:25:25.972704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:25:25.972989 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:25:25.973334 (+ 345us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:25:25.988888 (+ 15554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:25.990095 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.010893 (+ 20798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.012012 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.015528 (+ 3516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:26.015740 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 12:25:26.015820 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 12:25:26.015937 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7852,"thread_start_us":341,"threads_started":1}
I20260806 12:25:26.036487 14935 tablet_service.cc:1467] Processing CreateTablet for tablet 5c74645225f04889b49211d7deba3836 (DEFAULT_TABLE table=test-table [id=912b813ba7614bf2a9ec9f8a75ecd245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:25:26.038395 14935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c74645225f04889b49211d7deba3836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:25:26.056754 15062 tablet_bootstrap.cc:492] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43: Bootstrap starting.
I20260806 12:25:26.062075 15062 tablet_bootstrap.cc:654] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43: Neither blocks nor log segments found. Creating new log.
I20260806 12:25:26.062853 15064 tablet_bootstrap.cc:492] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7: Bootstrap starting.
I20260806 12:25:26.071242 15062 log.cc:824] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43: Log is configured to *not* fsync() on all Append() calls
I20260806 12:25:26.071736 15064 tablet_bootstrap.cc:654] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7: Neither blocks nor log segments found. Creating new log.
I20260806 12:25:26.073351 15064 log.cc:824] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7: Log is configured to *not* fsync() on all Append() calls
I20260806 12:25:26.076535 15065 tablet_bootstrap.cc:492] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa: Bootstrap starting.
I20260806 12:25:26.081903 15065 tablet_bootstrap.cc:654] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa: Neither blocks nor log segments found. Creating new log.
I20260806 12:25:26.084923 15062 tablet_bootstrap.cc:492] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43: No bootstrap required, opened a new log
I20260806 12:25:26.085345 15062 ts_tablet_manager.cc:1397] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.000s
I20260806 12:25:26.095829 15064 tablet_bootstrap.cc:492] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7: No bootstrap required, opened a new log
I20260806 12:25:26.096328 15064 ts_tablet_manager.cc:1397] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.005s
I20260806 12:25:26.105742 15065 log.cc:824] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa: Log is configured to *not* fsync() on all Append() calls
I20260806 12:25:26.107429 15062 raft_consensus.cc:348] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 43947 } } peers { permanent_uuid: "1be586c87a964245b8433f44fe10ce43" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 43123 } } peers { permanent_uuid: "28b882660c37408e97e4834424ad68fa" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36713 } }
I20260806 12:25:26.108906 15062 raft_consensus.cc:374] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:25:26.109087 15062 raft_consensus.cc:729] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be586c87a964245b8433f44fe10ce43, State: Initialized, Role: FOLLOWER
I20260806 12:25:26.110126 15062 consensus_queue.cc:260] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 43947 } } peers { permanent_uuid: "1be586c87a964245b8433f44fe10ce43" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 43123 } } peers { permanent_uuid: "28b882660c37408e97e4834424ad68fa" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36713 } }
I20260806 12:25:26.112059 15062 ts_tablet_manager.cc:1428] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43: Time spent starting tablet: real 0.027s user 0.005s sys 0.005s
I20260806 12:25:26.112892 15064 raft_consensus.cc:348] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 43947 } } peers { permanent_uuid: "1be586c87a964245b8433f44fe10ce43" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 43123 } } peers { permanent_uuid: "28b882660c37408e97e4834424ad68fa" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36713 } }
I20260806 12:25:26.113305 15064 raft_consensus.cc:374] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:25:26.120230 15064 raft_consensus.cc:729] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d4bcf663ab40a0bf84bc9bd7b10fe7, State: Initialized, Role: FOLLOWER
I20260806 12:25:26.121083 15064 consensus_queue.cc:260] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 43947 } } peers { permanent_uuid: "1be586c87a964245b8433f44fe10ce43" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 43123 } } peers { permanent_uuid: "28b882660c37408e97e4834424ad68fa" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36713 } }
I20260806 12:25:26.122516 14830 heartbeater.cc:503] Master 127.28.210.126:42373 was elected leader, sending a full tablet report...
I20260806 12:25:26.123021 15064 ts_tablet_manager.cc:1428] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7: Time spent starting tablet: real 0.027s user 0.005s sys 0.005s
I20260806 12:25:26.126041 15065 tablet_bootstrap.cc:492] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa: No bootstrap required, opened a new log
I20260806 12:25:26.126330 15065 ts_tablet_manager.cc:1397] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.000s
I20260806 12:25:26.138198 15065 raft_consensus.cc:348] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 43947 } } peers { permanent_uuid: "1be586c87a964245b8433f44fe10ce43" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 43123 } } peers { permanent_uuid: "28b882660c37408e97e4834424ad68fa" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36713 } }
I20260806 12:25:26.139264 15065 raft_consensus.cc:374] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:25:26.139420 15065 raft_consensus.cc:729] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b882660c37408e97e4834424ad68fa, State: Initialized, Role: FOLLOWER
I20260806 12:25:26.140225 15065 consensus_queue.cc:260] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 43947 } } peers { permanent_uuid: "1be586c87a964245b8433f44fe10ce43" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 43123 } } peers { permanent_uuid: "28b882660c37408e97e4834424ad68fa" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36713 } }
I20260806 12:25:26.149111 15065 ts_tablet_manager.cc:1428] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa: Time spent starting tablet: real 0.023s user 0.004s sys 0.006s
I20260806 12:25:26.149709 15039 heartbeater.cc:503] Master 127.28.210.126:42373 was elected leader, sending a full tablet report...
W20260806 12:25:26.153330 14623 tablet.cc:2369] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 12:25:26 dist-test-slave-lgr5 krb5kdc[13610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019124, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:25:26.276793 15071 raft_consensus.cc:482] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:25:26.277120 15071 raft_consensus.cc:504] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 43947 } } peers { permanent_uuid: "1be586c87a964245b8433f44fe10ce43" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 43123 } } peers { permanent_uuid: "28b882660c37408e97e4834424ad68fa" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36713 } }
I20260806 12:25:26.279068 15071 leader_election.cc:288] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78d4bcf663ab40a0bf84bc9bd7b10fe7 (127.28.210.66:43947), 28b882660c37408e97e4834424ad68fa (127.28.210.67:36713)
I20260806 12:25:26.298393 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:26.280667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:36713 (local address 127.28.210.65:60073)
0806 12:25:26.293002 (+ 12335us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:26.293015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:25:26.293212 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:25:26.293500 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:25:26.293503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:25:26.293524 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:25:26.293847 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:26.293856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.295063 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.295066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:26.296708 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:26.296721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.297718 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.297722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:26.297846 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:26.297873 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 12:25:26.298057 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 12:25:26.298158 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12260,"mutex_wait_us":3,"spinlock_wait_cycles":32640,"thread_start_us":228,"threads_started":1}
I20260806 12:25:26.298408 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:26.281493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:49535 (local address 127.28.210.66:43947)
0806 12:25:26.281658 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 12:25:26.281663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:25:26.292687 (+ 11024us) server_negotiation.cc:407] Connection header received
0806 12:25:26.292858 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:25:26.292863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:25:26.292929 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:25:26.293030 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:25:26.294119 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.294872 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.296979 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.297652 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.297762 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:26.297813 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 12:25:26.298172 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 12:25:26.298265 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 12:25:26.299157 15061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:26.280827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:60073 (local address 127.28.210.67:36713)
0806 12:25:26.281006 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 12:25:26.281012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:25:26.293200 (+ 12188us) server_negotiation.cc:407] Connection header received
0806 12:25:26.293321 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:25:26.293324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:25:26.293378 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:25:26.293459 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:25:26.294004 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.294808 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.296873 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.297431 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.297577 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:26.297637 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 12:25:26.298854 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0806 12:25:26.299020 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 12:25:26.299001 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:26.279709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:43947 (local address 127.28.210.65:49535)
0806 12:25:26.292393 (+ 12684us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:26.292416 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 12:25:26.292724 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:25:26.293596 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:25:26.293600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:25:26.293643 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:25:26.293958 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:26.293970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.295134 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.295138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:26.296799 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:25:26.296813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.297917 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.297920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:25:26.298026 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:26.298050 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 12:25:26.298749 (+ 699us) client_negotiation.cc:240] Negotiation successful
0806 12:25:26.298841 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12554,"thread_start_us":475,"threads_started":1}
I20260806 12:25:26.300420 14759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c74645225f04889b49211d7deba3836" candidate_uuid: "1be586c87a964245b8433f44fe10ce43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7" is_pre_election: true
I20260806 12:25:26.300426 14965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c74645225f04889b49211d7deba3836" candidate_uuid: "1be586c87a964245b8433f44fe10ce43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b882660c37408e97e4834424ad68fa" is_pre_election: true
I20260806 12:25:26.300786 14759 raft_consensus.cc:2454] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1be586c87a964245b8433f44fe10ce43 in term 0.
I20260806 12:25:26.300786 14965 raft_consensus.cc:2454] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1be586c87a964245b8433f44fe10ce43 in term 0.
I20260806 12:25:26.301465 14456 leader_election.cc:302] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be586c87a964245b8433f44fe10ce43, 78d4bcf663ab40a0bf84bc9bd7b10fe7; no voters:
I20260806 12:25:26.301878 15071 raft_consensus.cc:2781] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:25:26.301967 15071 raft_consensus.cc:482] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:25:26.302011 15071 raft_consensus.cc:3037] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:25:26.305928 15071 raft_consensus.cc:504] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 43947 } } peers { permanent_uuid: "1be586c87a964245b8433f44fe10ce43" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 43123 } } peers { permanent_uuid: "28b882660c37408e97e4834424ad68fa" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36713 } }
I20260806 12:25:26.306432 15071 leader_election.cc:288] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [CANDIDATE]: Term 1 election: Requested vote from peers 78d4bcf663ab40a0bf84bc9bd7b10fe7 (127.28.210.66:43947), 28b882660c37408e97e4834424ad68fa (127.28.210.67:36713)
I20260806 12:25:26.307221 14759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c74645225f04889b49211d7deba3836" candidate_uuid: "1be586c87a964245b8433f44fe10ce43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7"
I20260806 12:25:26.307402 14759 raft_consensus.cc:3037] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:25:26.309160 14759 raft_consensus.cc:2454] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be586c87a964245b8433f44fe10ce43 in term 1.
I20260806 12:25:26.310253 14965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c74645225f04889b49211d7deba3836" candidate_uuid: "1be586c87a964245b8433f44fe10ce43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b882660c37408e97e4834424ad68fa"
I20260806 12:25:26.310436 14965 raft_consensus.cc:3037] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:25:26.312594 14965 raft_consensus.cc:2454] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be586c87a964245b8433f44fe10ce43 in term 1.
I20260806 12:25:26.313383 14456 leader_election.cc:302] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be586c87a964245b8433f44fe10ce43, 78d4bcf663ab40a0bf84bc9bd7b10fe7; no voters:
I20260806 12:25:26.313944 15071 raft_consensus.cc:2781] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:25:26.314348 15071 raft_consensus.cc:686] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [term 1 LEADER]: Becoming Leader. State: Replica: 1be586c87a964245b8433f44fe10ce43, State: Running, Role: LEADER
I20260806 12:25:26.314826 15071 consensus_queue.cc:237] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 43947 } } peers { permanent_uuid: "1be586c87a964245b8433f44fe10ce43" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 43123 } } peers { permanent_uuid: "28b882660c37408e97e4834424ad68fa" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36713 } }
I20260806 12:25:26.319909 14307 catalog_manager.cc:5495] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1be586c87a964245b8433f44fe10ce43 (127.28.210.65). New cstate: current_term: 1 leader_uuid: "1be586c87a964245b8433f44fe10ce43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 43947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1be586c87a964245b8433f44fe10ce43" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 43123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28b882660c37408e97e4834424ad68fa" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36713 } health_report { overall_health: UNKNOWN } } }
I20260806 12:25:26.345989 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:26.339727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.28.210.65:43123)
0806 12:25:26.339878 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 12:25:26.339883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:25:26.339900 (+ 17us) server_negotiation.cc:407] Connection header received
0806 12:25:26.340036 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:25:26.340039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:25:26.340154 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:25:26.340296 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:25:26.340790 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.341634 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.342954 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:25:26.343359 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:25:26.343458 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:25:26.343936 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:25:26.344770 (+ 834us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:25:26.344908 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 12:25:26.345647 (+ 739us) server_negotiation.cc:299] Negotiation successful
0806 12:25:26.345737 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 12:25:26.356926 14965 raft_consensus.cc:1261] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa [term 1 FOLLOWER]: Refusing update from remote peer 1be586c87a964245b8433f44fe10ce43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:25:26.358222 15081 consensus_queue.cc:1035] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b882660c37408e97e4834424ad68fa" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 12:25:26.364262 14831 tablet.cc:2369] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:25:26.366315 14759 raft_consensus.cc:1261] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7 [term 1 FOLLOWER]: Refusing update from remote peer 1be586c87a964245b8433f44fe10ce43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:25:26.367295 15081 consensus_queue.cc:1035] T 5c74645225f04889b49211d7deba3836 P 1be586c87a964245b8433f44fe10ce43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78d4bcf663ab40a0bf84bc9bd7b10fe7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 43947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 12:25:26.374619 15040 tablet.cc:2369] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:25:26.380043 15086 mvcc.cc:205] Tried to move back new op lower bound from 7315534341545209856 to 7315534341391007744. Current Snapshot: MvccSnapshot[applied={T|T < 7315534341545209856}]
I20260806 12:25:26.386860 15087 mvcc.cc:205] Tried to move back new op lower bound from 7315534341545209856 to 7315534341391007744. Current Snapshot: MvccSnapshot[applied={T|T < 7315534341545209856}]
I20260806 12:25:26.394886 15091 mvcc.cc:205] Tried to move back new op lower bound from 7315534341545209856 to 7315534341391007744. Current Snapshot: MvccSnapshot[applied={T|T < 7315534341545209856}]
I20260806 12:25:26.399099 14307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:25:26.399314 14307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:25:26.402637 14307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ca14e98415748728421370d462eee86: Sending DeleteTablet for 3 replicas of tablet 5c74645225f04889b49211d7deba3836
I20260806 12:25:26.404966 14727 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c74645225f04889b49211d7deba3836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:25:26 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:38026
I20260806 12:25:26.406172 14530 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c74645225f04889b49211d7deba3836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:25:26 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:53804
I20260806 12:25:26.407069 14935 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c74645225f04889b49211d7deba3836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:25:26 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:43572
I20260806 12:25:26.408244 15101 tablet_replica.cc:330] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7: stopping tablet replica
I20260806 12:25:26.408773 15101 raft_consensus.cc:2229] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:25:26.409364 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 14425
I20260806 12:25:26.409417 15101 raft_consensus.cc:2258] T 5c74645225f04889b49211d7deba3836 P 78d4bcf663ab40a0bf84bc9bd7b10fe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:25:26.439558 15104 tablet_replica.cc:330] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa: stopping tablet replica
I20260806 12:25:26.440161 15104 raft_consensus.cc:2229] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 12:25:26.440361 14289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:43123 (error 108)
I20260806 12:25:26.440761 15104 raft_consensus.cc:2258] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:25:26.441427 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 14634
I20260806 12:25:26.467445 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:26.466696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:43123 (local address 127.0.0.1:53822)
0806 12:25:26.467239 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:26.467335 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444}
W20260806 12:25:26.467851 14289 catalog_manager.cc:4584] TS 1be586c87a964245b8433f44fe10ce43 (127.28.210.65:43123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c74645225f04889b49211d7deba3836: Network error: Client connection negotiation failed: client connection to 127.28.210.65:43123: connect: Connection refused (error 111)
I20260806 12:25:26.471513 15104 ts_tablet_manager.cc:1905] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:25:26.475747 15104 ts_tablet_manager.cc:1918] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:25:26.478394 15104 log.cc:1196] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/5c74645225f04889b49211d7deba3836
I20260806 12:25:26.478415 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:26.478012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:43947 (local address 127.0.0.1:38038)
0806 12:25:26.478235 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:26.478298 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.66:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260806 12:25:26.479099 15104 ts_tablet_manager.cc:1939] T 5c74645225f04889b49211d7deba3836 P 28b882660c37408e97e4834424ad68fa: Deleting consensus metadata
I20260806 12:25:26.481048 14286 catalog_manager.cc:4841] TS 28b882660c37408e97e4834424ad68fa (127.28.210.67:36713): tablet 5c74645225f04889b49211d7deba3836 (table test-table [id=912b813ba7614bf2a9ec9f8a75ecd245]) successfully deleted
I20260806 12:25:26.482234 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 14837
I20260806 12:25:26.559283 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:26.558732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:43123 (local address 127.0.0.1:53838)
0806 12:25:26.559108 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:26.559171 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
I20260806 12:25:26.559558 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:26.558950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:43947 (local address 127.0.0.1:38054)
0806 12:25:26.559463 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:26.559510 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.66:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467}
I20260806 12:25:26.561401 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:26.560878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:43123 (local address 127.0.0.1:53848)
0806 12:25:26.561199 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:26.561244 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261}
I20260806 12:25:26.561699 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:25:26.561066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:43947 (local address 127.0.0.1:38068)
0806 12:25:26.561591 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 12:25:26.561646 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.66:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465}
I20260806 12:25:26.575660 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 14257
I20260806 12:25:26.591348 29513 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 12:25:26.947444 29513 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 12:25:26.947738 29513 mini_ranger.cc:66] Stopping Ranger...
I20260806 12:25:27.294334 29513 mini_ranger.cc:68] Stopped Ranger
2026-08-06 12:25:27.294 UTC [13655] LOG: received smart shutdown request
2026-08-06 12:25:27.299 UTC [13655] LOG: background worker "logical replication launcher" (PID 13667) exited with exit code 1
2026-08-06 12:25:27.299 UTC [13661] LOG: shutting down
2026-08-06 12:25:27.532 UTC [13655] LOG: database system is shut down
2026-08-06T12:25:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (58861 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:25:27 dist-test-slave-lgr5 krb5kdc[15122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:25:27 dist-test-slave-lgr5 krb5kdc[15122](info): set up 2 sockets
Aug 06 12:25:27 dist-test-slave-lgr5 krb5kdc[15122](info): commencing operation
krb5kdc: starting...
W20260806 12:25:28.904191 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:25:29 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:25:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:25:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/rangeradmin_127.28.210.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/rangeradmin_127.28.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/rangerlookup_127.28.210.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/rangerlookup_127.28.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/HTTP_127.28.210.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/HTTP_127.28.210.84@KRBTEST.COM.keytab.
I20260806 12:25:29.123893 29513 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPswvRx/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/postgres -l logfile start
2026-08-06 12:25:32.089 UTC [15181] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 12:25:32.089 UTC [15181] LOG: listening on IPv4 address "127.28.210.84", port 37899
2026-08-06 12:25:32.094 UTC [15181] LOG: listening on Unix socket "/tmp/.s.PGSQL.37899"
2026-08-06 12:25:32.125 UTC [15185] LOG: database system was shut down at 2026-08-06 12:25:30 UTC
2026-08-06 12:25:32.128 UTC [15181] LOG: database system is ready to accept connections
I20260806 12:25:33.209569 29513 mini_postgres.cc:96] Postgres bound to 37899
2026-08-06 12:25:33.217 UTC [15197] FATAL: database "slave" does not exist
127.28.210.84:37899 - accepting connections
I20260806 12:25:33.218508 29513 mini_ranger.cc:161] Starting Ranger...
I20260806 12:25:33.237377 29513 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 12:25:33.809029 29513 mini_ranger.cc:87] Created ranger Postgres database
I20260806 12:25:33.809113 29513 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-admin
2026-08-06 12:25:34,124 [I] DB FLAVOR :POSTGRES
2026-08-06 12:25:34,125 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 12:25:34,125 [I] Checking connection..
2026-08-06 12:25:34,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 12:25:34,409 [I] Checking connection passed.
2026-08-06 12:25:34,410 [I] --------- Verifying version history table ---------
2026-08-06 12:25:34,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 12:25:34,698 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 12:25:34,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 12:25:34,971 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 12:25:34,971 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 12:25:34,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 12:25:35.267 UTC [15284] WARNING: there is no transaction in progress
2026-08-06 12:25:35,276 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 12:25:35,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 12:25:35,574 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 12:25:35,574 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 12:25:35,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 12:25:35,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 12:25:36,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lgr5.c.gcp-upstream.internal','N') ;"
2026-08-06 12:25:36,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 12:25:36,707 [I] Table x_portal_user does not exist in database ranger
2026-08-06 12:25:36,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 12:25:37,019 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 12:25:37,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 12:25:37,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 12:25:37,616 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 12:25:37,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 12:25:38.753 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:38.772 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:38.792 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:38.811 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:38.831 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:38.853 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:38.921 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:38.929 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:38.946 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:38.957 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:38.968 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:38.984 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:38.996 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:39.005 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:39.013 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:39.024 UTC [15435] WARNING: there is no transaction in progress
2026-08-06 12:25:39,817 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 12:25:39,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 12:25:40,095 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 12:25:40,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 12:25:40,378 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 12:25:40,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 12:25:40,662 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 12:25:40,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 12:25:40,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 12:25:41,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lgr5.c.gcp-upstream.internal';"
2026-08-06 12:25:41,519 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 12:25:41,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 12:25:41,816 [I] DB_PATCHES have already been applied
I20260806 12:25:41.823443 29513 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPswvRx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 12:25:41.823530 29513 mini_ranger.cc:193] Using host: 127.28.210.84
I20260806 12:25:41.829387 29513 mini_ranger.cc:227] Ranger admin URL: http://127.28.210.84:45663
Aug 06, 2026 12:25:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 12:25:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 12:25:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 12:25:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 12:25:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 12:25:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 12:25:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 12:25:43.106789 29513 mini_ranger.cc:160] Time spent starting Ranger: real 9.888s user 0.000s sys 0.014s
Aug 06, 2026 12:26:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 12:26:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 12:26:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 12:26:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 12:26:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 12:26:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 12:26:05.887319 29513 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/rangerkms_127.28.210.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/rangerkms_127.28.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.210.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.210.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.210.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/HTTP_127.28.210.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.210.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/HTTP_127.28.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 12:26:05 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019165, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 12:26:05.984781 29513 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 12:26:05.996562 29513 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 12:26:06.903200 29513 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 12:26:06.903273 29513 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-kms
I20260806 12:26:06.903322 29513 mini_ranger_kms.cc:227] Using postgres at 127.28.210.84:37899
2026-08-06 12:26:07,079 [I] DB FLAVOR :POSTGRES
2026-08-06 12:26:07,079 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 12:26:07,079 [I] Checking connection
2026-08-06 12:26:07,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 12:26:07,334 [I] connection success
2026-08-06 12:26:07,334 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 12:26:07,334 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 12:26:07,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 12:26:07,576 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 12:26:07,577 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 12:26:07,577 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 12:26:07,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.210.84:37899/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 12:26:07,864 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 12:26:08.358677 29513 mini_ranger_kms.cc:310] Created kms service
I20260806 12:26:08.580195 29513 mini_ranger_kms.cc:326] Created kudu user
I20260806 12:26:08.627246 29513 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 12:26:08 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019165, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:26:09.049523 29513 mini_ranger_kms.cc:384] Added ranger policy
I20260806 12:26:09.049809 29513 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPswvRx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPswvRx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 12:26:09.049839 29513 mini_ranger_kms.cc:240] Using host: 127.28.210.84
I20260806 12:26:09.054380 29513 mini_ranger_kms.cc:276] Ranger KMS PID: 15706
I20260806 12:26:09.054474 29513 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.210.84:47579
Aug 06, 2026 12:26:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 12:26:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 12:26:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 12:26:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 12:26:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 12:26:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 12:26:11 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786019171, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.210.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:11.578156 29513 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.593s user 0.003s sys 0.158s
I20260806 12:26:11.578305 29513 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 12:26:11.578385 29513 mini_ranger_kms.cc:401] 127.28.210.84:47579/kms/v1/keys
Aug 06 12:26:11 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019171, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.210.84@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
Aug 06 12:26:12 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019171, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.210.84@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
Aug 06 12:26:14 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:26:14.644616 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:36255
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:36255
--ranger_config_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.210.84:47579/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:14.764238 15794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:14.764526 15794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:14.764566 15794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:14.768576 15794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:26:14.768644 15794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:14.768661 15794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:14.768677 15794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:26:14.768690 15794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:26:14.774281 15794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.210.84:47579/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:36255
--ranger_config_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:36255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:14.775419 15794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:14.776515 15794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:14.785534 15801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:14.785645 15794 server_base.cc:1034] running on GCE node
W20260806 12:26:14.785497 15803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:14.785816 15808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:14.786422 15794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:14.787670 15794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:14.788847 15794 hybrid_clock.cc:648] HybridClock initialized: now 1786019174788820 us; error 44 us; skew 500 ppm
Aug 06 12:26:14 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:14.795698 15794 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:26:14.796859 15794 webserver.cc:460] Webserver started at http://127.28.210.126:42469/ using document root <none> and password file <none>
I20260806 12:26:14.797441 15794 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:14.797501 15794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:14.797712 15794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 12:26:14 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:26:14.847234 15794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "c02f2bbb889945a988e53ec6c13da4dd"
format_stamp: "Formatted at 2026-08-06 12:26:14 on dist-test-slave-lgr5"
server_key: "aae590ada16cbff50daeb7c275773706"
server_key_iv: "a9fbf9cdf86a0d7cb97a9525fd755a03"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:14.848122 15794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "c02f2bbb889945a988e53ec6c13da4dd"
format_stamp: "Formatted at 2026-08-06 12:26:14 on dist-test-slave-lgr5"
server_key: "aae590ada16cbff50daeb7c275773706"
server_key_iv: "a9fbf9cdf86a0d7cb97a9525fd755a03"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:14.853708 15794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 12:26:14.872520 15822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:14.874356 15794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 12:26:14.874545 15794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "c02f2bbb889945a988e53ec6c13da4dd"
format_stamp: "Formatted at 2026-08-06 12:26:14 on dist-test-slave-lgr5"
server_key: "aae590ada16cbff50daeb7c275773706"
server_key_iv: "a9fbf9cdf86a0d7cb97a9525fd755a03"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:14.874704 15794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:14.899027 15794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:14.902899 15794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:14.903141 15794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:14.912019 15794 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:36255
I20260806 12:26:14.912312 15901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:36255 every 8 connection(s)
I20260806 12:26:14.913134 15794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:26:14.913673 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 15794
I20260806 12:26:14.913794 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:26:14.914105 29513 external_mini_cluster.cc:1428] Setting key 80cfba878b4695df27849de85f5d1d2c
I20260806 12:26:14.917044 15903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 12:26:14 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:14.925266 15903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd: Bootstrap starting.
I20260806 12:26:14.928227 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:14.915652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.28.210.126:36255)
0806 12:26:14.916125 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:14.916145 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:14.916188 (+ 43us) server_negotiation.cc:407] Connection header received
0806 12:26:14.916855 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:14.916872 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:14.917249 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:14.917584 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:14.918397 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:14.919440 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:14.920315 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:14.920788 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:14.920946 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:14.923473 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:14.923480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:14.923497 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:14.923533 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:14.925155 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:14.925697 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:14.925703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:14.925709 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:14.925828 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:14.926111 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:14.926114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:14.926115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:14.926574 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:14.926777 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:14.927074 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 12:26:14.927395 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":122,"threads_started":1}
I20260806 12:26:14.928438 15903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:14.929916 15903 log.cc:824] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:14.932279 15903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd: No bootstrap required, opened a new log
I20260806 12:26:14.935439 15903 raft_consensus.cc:348] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02f2bbb889945a988e53ec6c13da4dd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36255 } }
I20260806 12:26:14.935705 15903 raft_consensus.cc:374] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:14.935742 15903 raft_consensus.cc:729] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c02f2bbb889945a988e53ec6c13da4dd, State: Initialized, Role: FOLLOWER
I20260806 12:26:14.936271 15903 consensus_queue.cc:260] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02f2bbb889945a988e53ec6c13da4dd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36255 } }
I20260806 12:26:14.936409 15903 raft_consensus.cc:388] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:26:14.936473 15903 raft_consensus.cc:482] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:26:14.936571 15903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:14.937606 15903 raft_consensus.cc:504] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02f2bbb889945a988e53ec6c13da4dd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36255 } }
I20260806 12:26:14.937981 15903 leader_election.cc:302] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c02f2bbb889945a988e53ec6c13da4dd; no voters:
I20260806 12:26:14.938342 15903 leader_election.cc:288] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:26:14.938699 15912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:26:14.938927 15912 raft_consensus.cc:686] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 1 LEADER]: Becoming Leader. State: Replica: c02f2bbb889945a988e53ec6c13da4dd, State: Running, Role: LEADER
I20260806 12:26:14.939266 15912 consensus_queue.cc:237] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02f2bbb889945a988e53ec6c13da4dd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36255 } }
I20260806 12:26:14.939713 15903 sys_catalog.cc:564] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:26:14.940615 15912 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c02f2bbb889945a988e53ec6c13da4dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02f2bbb889945a988e53ec6c13da4dd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36255 } } }
I20260806 12:26:14.940824 15912 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:14.941092 15912 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c02f2bbb889945a988e53ec6c13da4dd. Latest consensus state: current_term: 1 leader_uuid: "c02f2bbb889945a988e53ec6c13da4dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02f2bbb889945a988e53ec6c13da4dd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36255 } } }
I20260806 12:26:14.941231 15912 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:14.941534 15927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:26:14.941974 15903 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 12:26:14.948871 15927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:26:14.956892 15927 catalog_manager.cc:1347] Generated new cluster ID: 1e26cf4afd6e4cd3859df14970c6ccea
I20260806 12:26:14.956996 15927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:26:14.975318 15927 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:26:14.976809 15927 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:26:14.990041 15927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd: Generated new TSK 0
I20260806 12:26:14.991302 15927 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:26:15.763429 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.210.84:47579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36255
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:15.939374 15972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:15.939813 15972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:15.939949 15972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:15.946178 15972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:15.946266 15972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:15.946353 15972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:26:15.953846 15972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.210.84:47579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:15.955083 15972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:15.956554 15972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:15.970849 15980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:15.971005 15984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:15.978637 15981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:15.986471 15972 server_base.cc:1034] running on GCE node
I20260806 12:26:15.987205 15972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:15.988219 15972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:15.990103 15972 hybrid_clock.cc:648] HybridClock initialized: now 1786019175989987 us; error 118 us; skew 500 ppm
Aug 06 12:26:15 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:15.994424 15972 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:26:15.995806 15972 webserver.cc:460] Webserver started at http://127.28.210.65:40341/ using document root <none> and password file <none>
I20260806 12:26:15.996624 15972 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:15.996722 15972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:15.996958 15972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 12:26:15 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:26:16.015399 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "9d6864eee349464fa54c8421f889970f"
format_stamp: "Formatted at 2026-08-06 12:26:16 on dist-test-slave-lgr5"
server_key: "8142a7efb88ab350dd2f4f11e65ac36c"
server_key_iv: "73da163b4a798a66d3c3a06826090f6f"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:16.016528 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "9d6864eee349464fa54c8421f889970f"
format_stamp: "Formatted at 2026-08-06 12:26:16 on dist-test-slave-lgr5"
server_key: "8142a7efb88ab350dd2f4f11e65ac36c"
server_key_iv: "73da163b4a798a66d3c3a06826090f6f"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:16.022639 15972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260806 12:26:16.062398 15995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:16.065204 15972 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.000s
I20260806 12:26:16.065390 15972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "9d6864eee349464fa54c8421f889970f"
format_stamp: "Formatted at 2026-08-06 12:26:16 on dist-test-slave-lgr5"
server_key: "8142a7efb88ab350dd2f4f11e65ac36c"
server_key_iv: "73da163b4a798a66d3c3a06826090f6f"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:16.065565 15972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:16.103080 15972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:16.110322 15972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:16.110620 15972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:16.111459 15972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:16.112855 15972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:16.112958 15972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:16.113050 15972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:16.113106 15972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:16.125111 15972 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:45971
I20260806 12:26:16.126504 15972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:26:16.131563 16174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:45971 every 8 connection(s)
I20260806 12:26:16.133252 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 15972
I20260806 12:26:16.133380 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:26:16.133733 29513 external_mini_cluster.cc:1428] Setting key ab688dc592a0997af705653bcc70e946
Aug 06 12:26:16 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:26:16.156608 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:16.135474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:51615 (local address 127.28.210.126:36255)
0806 12:26:16.136745 (+ 1271us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:16.136750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:16.136767 (+ 17us) server_negotiation.cc:407] Connection header received
0806 12:26:16.137729 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:16.137732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:16.137793 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:16.137880 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:16.139613 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:16.140480 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:16.142367 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:16.142743 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:16.142825 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:16.150691 (+ 7866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:16.150696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:16.150699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:16.150743 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:16.152916 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:16.153860 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:16.153865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:16.153867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:16.153995 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:16.154533 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:16.154537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:16.154539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:16.154815 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:16.154921 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:16.156225 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0806 12:26:16.156393 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1192,"thread_start_us":93,"threads_started":1}
I20260806 12:26:16.157456 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:16.134125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36255 (local address 127.28.210.65:51615)
0806 12:26:16.135934 (+ 1809us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:16.136000 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:16.137327 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:16.138236 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:16.138247 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:16.138672 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:16.139421 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:16.139444 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:16.140643 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:16.140646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:16.142211 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:16.142225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:16.143076 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:16.143079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:16.143320 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:16.144046 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:16.144068 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:16.150323 (+ 6255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:16.153151 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:16.153161 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:16.153187 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:16.153695 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:16.154130 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:16.154133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:16.154135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:16.154387 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:16.154989 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:16.154998 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:16.155434 (+ 436us) client_negotiation.cc:769] Sending connection context
0806 12:26:16.155677 (+ 243us) client_negotiation.cc:240] Negotiation successful
0806 12:26:16.156120 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1569,"thread_start_us":261,"threads_started":1}
I20260806 12:26:16.159476 16176 heartbeater.cc:347] Connected to a master server at 127.28.210.126:36255
I20260806 12:26:16.159776 16176 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:16.160526 16176 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
I20260806 12:26:16.162777 15845 ts_manager.cc:194] Registered new tserver with Master: 9d6864eee349464fa54c8421f889970f (127.28.210.65:45971)
I20260806 12:26:16.164458 15845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:51615
Aug 06 12:26:16 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786019176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:26:16.178872 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.210.84:47579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36255
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:16.353526 16185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:16.353979 16185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:16.354046 16185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:16.360664 16185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:16.360816 16185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:16.360926 16185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:26:16.369520 16185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.210.84:47579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:16.371330 16185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:16.372951 16185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:16.386691 16199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:16.387871 16196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:16.388165 16197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:16.389036 16185 server_base.cc:1034] running on GCE node
I20260806 12:26:16.389741 16185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:16.390726 16185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:16.391935 16185 hybrid_clock.cc:648] HybridClock initialized: now 1786019176391909 us; error 54 us; skew 500 ppm
Aug 06 12:26:16 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:16.395862 16185 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:26:16.397087 16185 webserver.cc:460] Webserver started at http://127.28.210.66:41157/ using document root <none> and password file <none>
I20260806 12:26:16.397818 16185 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:16.397909 16185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:16.398159 16185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 12:26:16 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:26:16.415289 16185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "e2267e21c5f34498a58d65caeeb39392"
format_stamp: "Formatted at 2026-08-06 12:26:16 on dist-test-slave-lgr5"
server_key: "5654308224206e416edf70681a598bdb"
server_key_iv: "886dfd8f897c30c2b13a07770d3225f1"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:16.416108 16185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "e2267e21c5f34498a58d65caeeb39392"
format_stamp: "Formatted at 2026-08-06 12:26:16 on dist-test-slave-lgr5"
server_key: "5654308224206e416edf70681a598bdb"
server_key_iv: "886dfd8f897c30c2b13a07770d3225f1"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:16.420183 16185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:26:16.437367 16215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:16.438794 16185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:26:16.438933 16185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "e2267e21c5f34498a58d65caeeb39392"
format_stamp: "Formatted at 2026-08-06 12:26:16 on dist-test-slave-lgr5"
server_key: "5654308224206e416edf70681a598bdb"
server_key_iv: "886dfd8f897c30c2b13a07770d3225f1"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:16.439070 16185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:16.466547 16185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:16.470444 16185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:16.470731 16185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:16.471671 16185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:16.472949 16185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:16.473057 16185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:16.473157 16185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:16.473222 16185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:16.487814 16185 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:37521
I20260806 12:26:16.489171 16185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:26:16.490459 16386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:37521 every 8 connection(s)
I20260806 12:26:16.491816 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 16185
I20260806 12:26:16.491951 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:26:16.492322 29513 external_mini_cluster.cc:1428] Setting key 7c7e1aa80e0a446b44f55a423073a1f1
Aug 06 12:26:16 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:16.513509 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:16.494771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:50721 (local address 127.28.210.126:36255)
0806 12:26:16.494941 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:16.494946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:16.495382 (+ 436us) server_negotiation.cc:407] Connection header received
0806 12:26:16.496796 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:16.496801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:16.496863 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:16.496921 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:16.498856 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:16.499739 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:16.501619 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:16.501972 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:16.502059 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:16.509050 (+ 6991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:16.509055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:16.509058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:16.509092 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:16.510822 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:16.511528 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:16.511532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:16.511534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:16.511625 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:16.512048 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:16.512052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:16.512053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:16.512324 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:16.512420 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:16.513146 (+ 726us) server_negotiation.cc:299] Negotiation successful
0806 12:26:16.513295 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 12:26:16.515542 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:16.493459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36255 (local address 127.28.210.66:50721)
0806 12:26:16.495203 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:16.495274 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:16.496502 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:16.497450 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:16.497461 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:16.497868 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:16.498642 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:16.498666 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:16.499932 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:16.499935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:16.501459 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:16.501472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:16.502304 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:16.502309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:16.502548 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:16.506896 (+ 4348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:16.506924 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:16.508787 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:16.510992 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:16.510999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:16.511019 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:16.511385 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:16.511736 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:16.511739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:16.511741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:16.511933 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:16.512494 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:16.512501 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:16.512849 (+ 348us) client_negotiation.cc:769] Sending connection context
0806 12:26:16.514021 (+ 1172us) client_negotiation.cc:240] Negotiation successful
0806 12:26:16.514413 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1477,"thread_start_us":173,"threads_started":1}
I20260806 12:26:16.517307 16387 heartbeater.cc:347] Connected to a master server at 127.28.210.126:36255
I20260806 12:26:16.517589 16387 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:16.518244 16387 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
I20260806 12:26:16.519594 15845 ts_manager.cc:194] Registered new tserver with Master: e2267e21c5f34498a58d65caeeb39392 (127.28.210.66:37521)
I20260806 12:26:16.520423 15845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:50721
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:26:16.564980 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.210.84:47579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36255
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 12:26:16 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
W20260806 12:26:16.805429 16395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:16.805718 16395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:16.805780 16395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:16.810039 16395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:16.810103 16395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:16.810181 16395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:26:16.816015 16395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.210.84:47579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:16.817209 16395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:16.818274 16395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:16.828344 16409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:16.828509 16405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:16.828776 16407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:16.828817 16395 server_base.cc:1034] running on GCE node
I20260806 12:26:16.832742 16395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:16.833562 16395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:16.836182 16395 hybrid_clock.cc:648] HybridClock initialized: now 1786019176836145 us; error 35 us; skew 500 ppm
Aug 06 12:26:16 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:16.845044 16395 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:26:16.846292 16395 webserver.cc:460] Webserver started at http://127.28.210.67:34177/ using document root <none> and password file <none>
I20260806 12:26:16.847069 16395 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:16.847220 16395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:16.847512 16395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 12:26:16 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:26:16.863865 16395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7"
format_stamp: "Formatted at 2026-08-06 12:26:16 on dist-test-slave-lgr5"
server_key: "a821bccdccf490d1aa7553e70e9e7d65"
server_key_iv: "ac87cc030732e06c39e70387a80092ea"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:16.864938 16395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7"
format_stamp: "Formatted at 2026-08-06 12:26:16 on dist-test-slave-lgr5"
server_key: "a821bccdccf490d1aa7553e70e9e7d65"
server_key_iv: "ac87cc030732e06c39e70387a80092ea"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:16.870327 16395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 12:26:16.897738 16419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:16.899526 16395 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 12:26:16.899668 16395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7"
format_stamp: "Formatted at 2026-08-06 12:26:16 on dist-test-slave-lgr5"
server_key: "a821bccdccf490d1aa7553e70e9e7d65"
server_key_iv: "ac87cc030732e06c39e70387a80092ea"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:16.899809 16395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:16.922005 16395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:16.925825 16395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:16.926092 16395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:16.926889 16395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:16.928113 16395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:16.928228 16395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:16.928331 16395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:16.928395 16395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:16.948052 16395 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:40277
I20260806 12:26:16.951333 16589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:40277 every 8 connection(s)
I20260806 12:26:16.953266 16395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:26:16.958833 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 16395
I20260806 12:26:16.958976 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:26:16.959277 29513 external_mini_cluster.cc:1428] Setting key 820b96e7e6debafb805f79cd24b4574f
Aug 06 12:26:16 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:16.983132 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:16.958484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:37967 (local address 127.28.210.126:36255)
0806 12:26:16.958621 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:16.958626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:16.959239 (+ 613us) server_negotiation.cc:407] Connection header received
0806 12:26:16.960890 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:16.960894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:16.960970 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:16.961030 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:16.963104 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:16.963962 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:16.965934 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:16.966307 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:16.966389 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:16.973689 (+ 7300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:16.973693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:16.973697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:16.973733 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:16.975756 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:16.978560 (+ 2804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:16.978566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:16.978568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:16.978690 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:16.979210 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:16.979214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:16.979215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:16.979464 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:16.979549 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:16.982325 (+ 2776us) server_negotiation.cc:299] Negotiation successful
0806 12:26:16.982561 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 12:26:16.990877 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:16.957060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36255 (local address 127.28.210.67:37967)
0806 12:26:16.959045 (+ 1985us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:16.959113 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:16.960606 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:16.961502 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:16.961511 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:16.962075 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:16.962896 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:16.962920 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:16.964183 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:16.964186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:16.965777 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:16.965790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:16.966630 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:16.966634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:16.966919 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:16.967982 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:16.968005 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:16.973450 (+ 5445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:16.976352 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:16.976362 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:16.976387 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:16.976882 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:16.978818 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:16.978821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:16.978823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:16.979086 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:16.979683 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:16.979691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:16.980266 (+ 575us) client_negotiation.cc:769] Sending connection context
0806 12:26:16.983419 (+ 3153us) client_negotiation.cc:240] Negotiation successful
0806 12:26:16.983834 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1704,"thread_start_us":320,"threads_started":1}
I20260806 12:26:16.992347 16591 heartbeater.cc:347] Connected to a master server at 127.28.210.126:36255
I20260806 12:26:16.992631 16591 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:16.993287 16591 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
I20260806 12:26:16.994625 15845 ts_manager.cc:194] Registered new tserver with Master: f741699fcac84ffeb78dbf0ac6cc6bd7 (127.28.210.67:40277)
I20260806 12:26:16.995420 15845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:37967
I20260806 12:26:16.997992 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:26:16.998977 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 15972
I20260806 12:26:17.021965 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 16185
I20260806 12:26:17.049209 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 16395
I20260806 12:26:17.094583 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 15794
I20260806 12:26:17.114969 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:36255
--webserver_interface=127.28.210.126
--webserver_port=42469
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:36255
--ranger_config_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.210.84:47579/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.210.84:47579/kms with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:17.236594 16596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:17.236958 16596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:17.237020 16596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:17.241360 16596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:26:17.241451 16596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:17.241477 16596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:17.241497 16596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:26:17.241528 16596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:26:17.247623 16596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.210.84:47579/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:36255
--ranger_config_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:36255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:17.248956 16596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:17.250124 16596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:17.259727 16607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:17.259905 16596 server_base.cc:1034] running on GCE node
W20260806 12:26:17.259774 16606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:17.259773 16609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:17.260749 16596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:17.261971 16596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:17.263165 16596 hybrid_clock.cc:648] HybridClock initialized: now 1786019177263124 us; error 51 us; skew 500 ppm
Aug 06 12:26:17 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:17.266237 16596 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:26:17.267406 16596 webserver.cc:460] Webserver started at http://127.28.210.126:42469/ using document root <none> and password file <none>
I20260806 12:26:17.268016 16596 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:17.268131 16596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 12:26:17 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:26:17.307168 16596 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:17.309854 16621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:17.311225 16596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 12:26:17.311378 16596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "c02f2bbb889945a988e53ec6c13da4dd"
format_stamp: "Formatted at 2026-08-06 12:26:14 on dist-test-slave-lgr5"
server_key: "aae590ada16cbff50daeb7c275773706"
server_key_iv: "a9fbf9cdf86a0d7cb97a9525fd755a03"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:17.311997 16596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:17.326100 16596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:17.329752 16596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:17.329963 16596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:17.337503 16596 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:36255
I20260806 12:26:17.337523 16695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:36255 every 8 connection(s)
I20260806 12:26:17.338909 16596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:26:17.343977 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 16596
I20260806 12:26:17.343998 16697 sys_catalog.cc:263] Verifying existing consensus state
I20260806 12:26:17.344484 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.210.84:47579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:45971
--local_ip_for_outbound_sockets=127.28.210.65
--tserver_master_addrs=127.28.210.126:36255
--webserver_port=40341
--webserver_interface=127.28.210.65
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.210.84:47579/kms with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:26:17.345938 16697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd: Bootstrap starting.
I20260806 12:26:17.362829 16697 log.cc:824] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:17.369932 16697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 12:26:17.370358 16697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd: Bootstrap complete.
I20260806 12:26:17.374656 16697 raft_consensus.cc:348] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02f2bbb889945a988e53ec6c13da4dd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36255 } }
I20260806 12:26:17.375135 16697 raft_consensus.cc:729] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c02f2bbb889945a988e53ec6c13da4dd, State: Initialized, Role: FOLLOWER
I20260806 12:26:17.375720 16697 consensus_queue.cc:260] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02f2bbb889945a988e53ec6c13da4dd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36255 } }
I20260806 12:26:17.375864 16697 raft_consensus.cc:388] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:26:17.375934 16697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:26:17.376039 16697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 1 FOLLOWER]: Advancing to term 2
I20260806 12:26:17.378319 16697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02f2bbb889945a988e53ec6c13da4dd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36255 } }
I20260806 12:26:17.378679 16697 leader_election.cc:302] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c02f2bbb889945a988e53ec6c13da4dd; no voters:
I20260806 12:26:17.379000 16697 leader_election.cc:288] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 12:26:17.379156 16703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 2 FOLLOWER]: Leader election won for term 2
I20260806 12:26:17.379508 16703 raft_consensus.cc:686] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [term 2 LEADER]: Becoming Leader. State: Replica: c02f2bbb889945a988e53ec6c13da4dd, State: Running, Role: LEADER
I20260806 12:26:17.379910 16703 consensus_queue.cc:237] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02f2bbb889945a988e53ec6c13da4dd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36255 } }
I20260806 12:26:17.380301 16697 sys_catalog.cc:564] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:26:17.381424 16708 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c02f2bbb889945a988e53ec6c13da4dd. Latest consensus state: current_term: 2 leader_uuid: "c02f2bbb889945a988e53ec6c13da4dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02f2bbb889945a988e53ec6c13da4dd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36255 } } }
I20260806 12:26:17.381552 16708 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:17.382259 16704 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c02f2bbb889945a988e53ec6c13da4dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02f2bbb889945a988e53ec6c13da4dd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36255 } } }
I20260806 12:26:17.382344 16704 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:17.382578 16697 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 12:26:17.382653 16718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:26:17.389209 16718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:26:17.389979 16718 catalog_manager.cc:1259] Loaded cluster ID: 1e26cf4afd6e4cd3859df14970c6ccea
I20260806 12:26:17.390208 16718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:26:17.393119 16718 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:26:17.394055 16718 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd: Loaded TSK: 0
I20260806 12:26:17.395404 16718 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 12:26:17.472662 16700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:17.473181 16700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:17.473278 16700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:17.479743 16700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:17.479875 16700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:17.480000 16700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:26:17.488245 16700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.210.84:47579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:45971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=40341
--tserver_master_addrs=127.28.210.126:36255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:17.489825 16700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:17.491206 16700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:17.504037 16756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:17.504033 16755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:17.504523 16760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:17.504930 16700 server_base.cc:1034] running on GCE node
I20260806 12:26:17.505450 16700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:17.506310 16700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:17.507522 16700 hybrid_clock.cc:648] HybridClock initialized: now 1786019177507483 us; error 55 us; skew 500 ppm
Aug 06 12:26:17 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:17.511173 16700 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:26:17.512490 16700 webserver.cc:460] Webserver started at http://127.28.210.65:40341/ using document root <none> and password file <none>
I20260806 12:26:17.513224 16700 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:17.513332 16700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 12:26:17 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:26:17.541816 16700 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.001s
I20260806 12:26:17.545528 16773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:17.547144 16700 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 12:26:17.547276 16700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "9d6864eee349464fa54c8421f889970f"
format_stamp: "Formatted at 2026-08-06 12:26:16 on dist-test-slave-lgr5"
server_key: "8142a7efb88ab350dd2f4f11e65ac36c"
server_key_iv: "73da163b4a798a66d3c3a06826090f6f"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:17.548005 16700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:17.563441 16700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:17.567166 16700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:17.567448 16700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:17.568291 16700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:17.569504 16700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:17.569612 16700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:17.569736 16700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:17.569813 16700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:17.581827 16700 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:45971
I20260806 12:26:17.583178 16700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:26:17.583776 16942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:45971 every 8 connection(s)
I20260806 12:26:17.592989 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 16700
I20260806 12:26:17.593444 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.210.84:47579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:37521
--local_ip_for_outbound_sockets=127.28.210.66
--tserver_master_addrs=127.28.210.126:36255
--webserver_port=41157
--webserver_interface=127.28.210.66
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.210.84:47579/kms with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 12:26:17 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:17.606837 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:17.585901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36255 (local address 127.28.210.65:51305)
0806 12:26:17.586568 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:17.586634 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:17.587759 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:17.589086 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:17.589095 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:17.589463 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:17.590158 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:17.590181 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:17.591500 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:17.591503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:17.592668 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:17.592683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:17.594060 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:17.594065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:17.594267 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:17.594960 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:17.594983 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:17.599162 (+ 4179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:17.602476 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:17.602486 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:17.602507 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:17.602941 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:17.603472 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:17.603475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:17.603477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:17.603689 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:17.604634 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:17.604644 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:17.605008 (+ 364us) client_negotiation.cc:769] Sending connection context
0806 12:26:17.605246 (+ 238us) client_negotiation.cc:240] Negotiation successful
0806 12:26:17.605652 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":135,"threads_started":1}
I20260806 12:26:17.608667 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:17.585725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:51305 (local address 127.28.210.126:36255)
0806 12:26:17.586368 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:17.586392 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:17.586779 (+ 387us) server_negotiation.cc:407] Connection header received
0806 12:26:17.588299 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:17.588314 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:17.588635 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:17.588913 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:17.590355 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:17.591354 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:17.592824 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:17.593601 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:17.593786 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:17.599538 (+ 5752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:17.599547 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:17.599569 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:17.599616 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:17.602240 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:17.603122 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:17.603130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:17.603136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:17.603333 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:17.603808 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:17.603812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:17.603814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:17.604486 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:17.604651 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:17.607436 (+ 2785us) server_negotiation.cc:299] Negotiation successful
0806 12:26:17.607742 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":127,"threads_started":1}
I20260806 12:26:17.611730 16943 heartbeater.cc:347] Connected to a master server at 127.28.210.126:36255
I20260806 12:26:17.611980 16943 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:17.612524 16943 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
W20260806 12:26:17.614737 16943 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:36255 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 12:26:17.614879 16943 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:17.615089 16943 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
I20260806 12:26:17.615972 16943 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:17.616231 16943 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
W20260806 12:26:17.617095 16943 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 12:26:17.744513 16948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:17.744794 16948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:17.744865 16948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:17.749092 16948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:17.749159 16948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:17.749243 16948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:26:17.755213 16948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.210.84:47579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:37521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=41157
--tserver_master_addrs=127.28.210.126:36255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:17.756630 16948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:17.757723 16948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:17.768162 16956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:17.768062 16955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:17.768592 16948 server_base.cc:1034] running on GCE node
W20260806 12:26:17.769554 16960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:17.770164 16948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:17.771170 16948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:17.772467 16948 hybrid_clock.cc:648] HybridClock initialized: now 1786019177772425 us; error 60 us; skew 500 ppm
Aug 06 12:26:17 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:17.776561 16948 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:26:17.778151 16948 webserver.cc:460] Webserver started at http://127.28.210.66:41157/ using document root <none> and password file <none>
I20260806 12:26:17.779047 16948 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:17.779187 16948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 12:26:17 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:26:17.804137 16948 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:26:17.807823 16973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:17.809736 16948 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:26:17.809911 16948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "e2267e21c5f34498a58d65caeeb39392"
format_stamp: "Formatted at 2026-08-06 12:26:16 on dist-test-slave-lgr5"
server_key: "5654308224206e416edf70681a598bdb"
server_key_iv: "886dfd8f897c30c2b13a07770d3225f1"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:17.810797 16948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:17.827005 16948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:17.830960 16948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:17.831238 16948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:17.832190 16948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:17.833549 16948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:17.833631 16948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:17.833698 16948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:17.833731 16948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:17.847786 16948 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:37521
I20260806 12:26:17.849733 16948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:26:17.851207 17137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:37521 every 8 connection(s)
I20260806 12:26:17.855682 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 16948
I20260806 12:26:17.856204 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.210.84:47579/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:40277
--local_ip_for_outbound_sockets=127.28.210.67
--tserver_master_addrs=127.28.210.126:36255
--webserver_port=34177
--webserver_interface=127.28.210.67
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.210.84:47579/kms with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 12:26:17 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:17.873256 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:17.854074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:45943 (local address 127.28.210.126:36255)
0806 12:26:17.854247 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:17.854253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:17.855387 (+ 1134us) server_negotiation.cc:407] Connection header received
0806 12:26:17.856934 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:17.856939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:17.857009 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:17.857075 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:17.859039 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:17.859862 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:17.861788 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:17.862040 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:17.862153 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:17.867785 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:17.867792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:17.867795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:17.867841 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:17.869972 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:17.870889 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:17.870894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:17.870896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:17.871013 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:17.871579 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:17.871583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:17.871585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:17.871878 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:17.871986 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:17.872881 (+ 895us) server_negotiation.cc:299] Negotiation successful
0806 12:26:17.873055 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 12:26:17.875237 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:17.854494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36255 (local address 127.28.210.66:45943)
0806 12:26:17.855150 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:17.855226 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:17.856613 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:17.857600 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:17.857612 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:17.858010 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:17.858806 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:17.858833 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:17.860071 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:17.860074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:17.861591 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:17.861605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:17.862218 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:17.862222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:17.862465 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:17.863226 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:17.863252 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:17.867430 (+ 4178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:17.870193 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:17.870202 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:17.870227 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:17.870722 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:17.871166 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:17.871170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:17.871172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:17.871432 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:17.872108 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:17.872117 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:17.872534 (+ 417us) client_negotiation.cc:769] Sending connection context
0806 12:26:17.873582 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0806 12:26:17.874001 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":153,"threads_started":1}
I20260806 12:26:17.877033 17141 heartbeater.cc:347] Connected to a master server at 127.28.210.126:36255
I20260806 12:26:17.877389 17141 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:17.878110 17141 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
W20260806 12:26:17.880208 17141 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:36255 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 12:26:17.880378 17141 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:17.880641 17141 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
I20260806 12:26:17.881786 17141 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:17.882026 17141 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
W20260806 12:26:17.883153 17141 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 12:26:18.016294 17145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:18.016845 17145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:18.017030 17145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:18.023751 17145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:18.023864 17145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:18.023963 17145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:26:18.032243 17145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.210.84:47579/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:40277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=34177
--tserver_master_addrs=127.28.210.126:36255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:18.033809 17145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:18.035274 17145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:18.050566 17161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:18.050544 17160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:18.051079 17164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:18.051294 17145 server_base.cc:1034] running on GCE node
I20260806 12:26:18.051774 17145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:18.052593 17145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:18.053900 17145 hybrid_clock.cc:648] HybridClock initialized: now 1786019178053856 us; error 58 us; skew 500 ppm
Aug 06 12:26:18 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:18.057489 17145 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:26:18.058684 17145 webserver.cc:460] Webserver started at http://127.28.210.67:34177/ using document root <none> and password file <none>
I20260806 12:26:18.059331 17145 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:18.059446 17145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 12:26:18 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:26:18.082257 17145 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:26:18.085496 17176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:18.087184 17145 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 12:26:18.087332 17145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7"
format_stamp: "Formatted at 2026-08-06 12:26:16 on dist-test-slave-lgr5"
server_key: "a821bccdccf490d1aa7553e70e9e7d65"
server_key_iv: "ac87cc030732e06c39e70387a80092ea"
server_key_version: "kuduclusterkey@0"
I20260806 12:26:18.087939 17145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:18.109686 17145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:18.113456 17145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:18.113749 17145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:18.114554 17145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:18.115712 17145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:18.115823 17145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:18.115931 17145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:18.115964 17145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:18.127138 17145 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:40277
I20260806 12:26:18.127173 17349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:40277 every 8 connection(s)
I20260806 12:26:18.128553 17145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:26:18.137986 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 17145
Aug 06 12:26:18 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:18.143205 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:18.130685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:47123 (local address 127.28.210.126:36255)
0806 12:26:18.130838 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:18.130842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:18.131955 (+ 1113us) server_negotiation.cc:407] Connection header received
0806 12:26:18.132969 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:18.132973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:18.133032 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:18.133108 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:18.134203 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:18.134738 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:18.135882 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:18.136151 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:18.136245 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:18.139326 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:18.139330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:18.139333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:18.139363 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:18.140968 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:18.141501 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:18.141504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:18.141506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:18.141591 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:18.141984 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:18.141988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:18.141990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:18.142214 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:18.142315 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:18.142913 (+ 598us) server_negotiation.cc:299] Negotiation successful
0806 12:26:18.143039 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 12:26:18.144174 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:18.131122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36255 (local address 127.28.210.67:47123)
0806 12:26:18.131695 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:18.131749 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:18.132755 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:18.133293 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:18.133301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:18.133570 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:18.134058 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:18.134071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:18.134860 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:18.134864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:18.135754 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:18.135764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:18.136295 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:18.136300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:18.136471 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:18.137028 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:18.137045 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:18.139117 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:18.141103 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:18.141110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:18.141127 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:18.141403 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:18.141688 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:18.141691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:18.141693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:18.141875 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:18.142372 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:18.142379 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:18.142669 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 12:26:18.142847 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 12:26:18.143178 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":212,"threads_started":1}
I20260806 12:26:18.145659 17350 heartbeater.cc:347] Connected to a master server at 127.28.210.126:36255
I20260806 12:26:18.145916 17350 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:18.146395 17350 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
W20260806 12:26:18.148728 17350 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:36255 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 12:26:18.148862 17350 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:18.149040 17350 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
I20260806 12:26:18.150166 17350 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:18.150359 17350 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
W20260806 12:26:18.151196 17350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 12:26:18.154402 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:18.138649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.28.210.126:36255)
0806 12:26:18.139807 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:18.139812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:18.139824 (+ 12us) server_negotiation.cc:407] Connection header received
0806 12:26:18.140140 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:18.140145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:18.140203 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:18.140306 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:18.141589 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:18.142348 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:18.145203 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:18.145522 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:18.145613 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:18.151387 (+ 5774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:18.151391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:18.151393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:18.151415 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:18.152765 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:18.153167 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:18.153171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:18.153172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:18.153244 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:18.153557 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:18.153561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:18.153562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:18.153740 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:18.153940 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:18.154095 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 12:26:18.154215 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049,"thread_start_us":97,"threads_started":1}
I20260806 12:26:18.617324 16943 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:18.617748 16943 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
I20260806 12:26:18.618953 16647 ts_manager.cc:194] Registered new tserver with Master: 9d6864eee349464fa54c8421f889970f (127.28.210.65:45971)
I20260806 12:26:18.620349 16647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:51305
Aug 06 12:26:18 dist-test-slave-lgr5 krb5kdc[15122](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786019178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:18.883476 17141 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:18.883810 17141 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
I20260806 12:26:18.884744 16647 ts_manager.cc:194] Registered new tserver with Master: e2267e21c5f34498a58d65caeeb39392 (127.28.210.66:37521)
I20260806 12:26:18.885552 16647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:45943
Aug 06 12:26:19 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
I20260806 12:26:19.151420 17350 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:19.151762 17350 heartbeater.cc:511] Master 127.28.210.126:36255 requested a full tablet report, sending...
I20260806 12:26:19.152842 16647 ts_manager.cc:194] Registered new tserver with Master: f741699fcac84ffeb78dbf0ac6cc6bd7 (127.28.210.67:40277)
I20260806 12:26:19.153738 16647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:47123
I20260806 12:26:19.161342 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:26:19.172859 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.163110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.28.210.126:36255)
0806 12:26:19.163390 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:19.163395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:19.163514 (+ 119us) server_negotiation.cc:407] Connection header received
0806 12:26:19.163652 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:19.163655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:19.163706 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:19.163801 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:19.164677 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.165488 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.166716 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.167089 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.167202 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.168882 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:19.168886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:19.168889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:19.168918 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:19.170658 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:19.171138 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:19.171142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:19.171143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:19.171250 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:19.171541 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:19.171544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:19.171546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:19.171757 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:19.171873 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:19.172555 (+ 682us) server_negotiation.cc:299] Negotiation successful
0806 12:26:19.172682 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
I20260806 12:26:19.178782 16647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:26:19.183696 16647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:26:19.205286 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.196832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.28.210.65:45971)
0806 12:26:19.197216 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:19.197223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:19.197850 (+ 627us) server_negotiation.cc:407] Connection header received
0806 12:26:19.198091 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:19.198099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:19.198345 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:19.198564 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:19.198869 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.200015 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.202507 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.203326 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.203431 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.203613 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:19.205004 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0806 12:26:19.205125 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":95,"threads_started":1}
I20260806 12:26:19.206038 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.196662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:45971 (local address 127.0.0.1:37950)
0806 12:26:19.197598 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:19.197610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:19.197926 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:19.198559 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:19.198565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:19.198590 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:19.198737 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.198743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.200973 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.200976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.202347 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.202359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.204646 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.204655 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.204780 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.204822 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 12:26:19.205758 (+ 936us) client_negotiation.cc:240] Negotiation successful
0806 12:26:19.205906 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":90,"threads_started":1}
I20260806 12:26:19.208178 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.197339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:40277 (local address 127.0.0.1:57700)
0806 12:26:19.200292 (+ 2953us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:19.200315 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:19.200421 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:19.201794 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:19.201797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:19.201818 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:19.201919 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.201925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.203372 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.203375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.205924 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.205937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.207783 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.207786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.207873 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.207897 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 12:26:19.207963 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 12:26:19.208034 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2882,"thread_start_us":93,"threads_started":1}
I20260806 12:26:19.208890 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.197510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.28.210.67:40277)
0806 12:26:19.197853 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:19.197860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:19.200533 (+ 2673us) server_negotiation.cc:407] Connection header received
0806 12:26:19.200622 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:19.200628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:19.200851 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:19.201005 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:19.202056 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.203225 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.206083 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.206939 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.207053 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.207205 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:19.208652 (+ 1447us) server_negotiation.cc:299] Negotiation successful
0806 12:26:19.208756 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":70,"threads_started":1}
I20260806 12:26:19.209786 16823 tablet_service.cc:1467] Processing CreateTablet for tablet 7405bf34b0e641b2ae0fab2b64951d45 (DEFAULT_TABLE table=test-table [id=18b3ae8771ff44ba8fac7f1e44538665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:26:19.211014 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7405bf34b0e641b2ae0fab2b64951d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:19.211740 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.195751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:37521 (local address 127.0.0.1:47744)
0806 12:26:19.197030 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:19.197088 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:19.197347 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:19.204197 (+ 6850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:19.204201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:19.204224 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:19.204337 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.204345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.209163 (+ 4818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.209167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.210271 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.210283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.211365 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.211369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.211470 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.211495 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 12:26:19.211540 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 12:26:19.211609 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"thread_start_us":109,"threads_started":1}
I20260806 12:26:19.211905 17223 tablet_service.cc:1467] Processing CreateTablet for tablet 7405bf34b0e641b2ae0fab2b64951d45 (DEFAULT_TABLE table=test-table [id=18b3ae8771ff44ba8fac7f1e44538665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:26:19.212951 17223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7405bf34b0e641b2ae0fab2b64951d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:19.213342 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.195922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.28.210.66:37521)
0806 12:26:19.201348 (+ 5426us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:19.201355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:19.201377 (+ 22us) server_negotiation.cc:407] Connection header received
0806 12:26:19.201454 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:19.201459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:19.201653 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:19.204522 (+ 2869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:19.204689 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.207652 (+ 2963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.210427 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.211225 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.211999 (+ 774us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.212180 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:19.212236 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 12:26:19.212329 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5299,"thread_start_us":255,"threads_started":1}
I20260806 12:26:19.216300 17043 tablet_service.cc:1467] Processing CreateTablet for tablet 7405bf34b0e641b2ae0fab2b64951d45 (DEFAULT_TABLE table=test-table [id=18b3ae8771ff44ba8fac7f1e44538665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:26:19.217393 17043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7405bf34b0e641b2ae0fab2b64951d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:19.223812 17405 tablet_bootstrap.cc:492] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7: Bootstrap starting.
I20260806 12:26:19.226557 17405 tablet_bootstrap.cc:654] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:19.227588 17405 log.cc:824] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:19.228556 17404 tablet_bootstrap.cc:492] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f: Bootstrap starting.
I20260806 12:26:19.230893 17405 tablet_bootstrap.cc:492] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7: No bootstrap required, opened a new log
I20260806 12:26:19.231081 17404 tablet_bootstrap.cc:654] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:19.231160 17405 ts_tablet_manager.cc:1397] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260806 12:26:19.232071 17404 log.cc:824] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:19.234973 17404 tablet_bootstrap.cc:492] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f: No bootstrap required, opened a new log
I20260806 12:26:19.235237 17404 ts_tablet_manager.cc:1397] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 12:26:19.235675 17405 raft_consensus.cc:348] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2267e21c5f34498a58d65caeeb39392" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37521 } } peers { permanent_uuid: "9d6864eee349464fa54c8421f889970f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 45971 } } peers { permanent_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40277 } }
I20260806 12:26:19.235971 17405 raft_consensus.cc:374] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:19.236033 17405 raft_consensus.cc:729] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f741699fcac84ffeb78dbf0ac6cc6bd7, State: Initialized, Role: FOLLOWER
I20260806 12:26:19.236614 17405 consensus_queue.cc:260] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2267e21c5f34498a58d65caeeb39392" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37521 } } peers { permanent_uuid: "9d6864eee349464fa54c8421f889970f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 45971 } } peers { permanent_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40277 } }
I20260806 12:26:19.237941 17405 ts_tablet_manager.cc:1428] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 12:26:19.238412 17350 heartbeater.cc:503] Master 127.28.210.126:36255 was elected leader, sending a full tablet report...
I20260806 12:26:19.239369 17404 raft_consensus.cc:348] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2267e21c5f34498a58d65caeeb39392" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37521 } } peers { permanent_uuid: "9d6864eee349464fa54c8421f889970f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 45971 } } peers { permanent_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40277 } }
I20260806 12:26:19.239670 17404 raft_consensus.cc:374] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:19.239763 17404 raft_consensus.cc:729] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6864eee349464fa54c8421f889970f, State: Initialized, Role: FOLLOWER
I20260806 12:26:19.240350 17404 consensus_queue.cc:260] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2267e21c5f34498a58d65caeeb39392" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37521 } } peers { permanent_uuid: "9d6864eee349464fa54c8421f889970f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 45971 } } peers { permanent_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40277 } }
I20260806 12:26:19.241503 17404 ts_tablet_manager.cc:1428] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 12:26:19.241899 16943 heartbeater.cc:503] Master 127.28.210.126:36255 was elected leader, sending a full tablet report...
I20260806 12:26:19.244890 17408 tablet_bootstrap.cc:492] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392: Bootstrap starting.
I20260806 12:26:19.247430 17408 tablet_bootstrap.cc:654] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:19.248330 17408 log.cc:824] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:19.251173 17408 tablet_bootstrap.cc:492] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392: No bootstrap required, opened a new log
I20260806 12:26:19.251384 17408 ts_tablet_manager.cc:1397] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 12:26:19.255286 17408 raft_consensus.cc:348] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2267e21c5f34498a58d65caeeb39392" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37521 } } peers { permanent_uuid: "9d6864eee349464fa54c8421f889970f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 45971 } } peers { permanent_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40277 } }
I20260806 12:26:19.255595 17408 raft_consensus.cc:374] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:19.255685 17408 raft_consensus.cc:729] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2267e21c5f34498a58d65caeeb39392, State: Initialized, Role: FOLLOWER
I20260806 12:26:19.256199 17408 consensus_queue.cc:260] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2267e21c5f34498a58d65caeeb39392" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37521 } } peers { permanent_uuid: "9d6864eee349464fa54c8421f889970f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 45971 } } peers { permanent_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40277 } }
I20260806 12:26:19.257364 17408 ts_tablet_manager.cc:1428] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 12:26:19.257845 17141 heartbeater.cc:503] Master 127.28.210.126:36255 was elected leader, sending a full tablet report...
W20260806 12:26:19.336197 16944 tablet.cc:2369] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 12:26:19 dist-test-slave-lgr5 krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for HTTP/127.28.210.84@KRBTEST.COM
W20260806 12:26:19.352254 17142 tablet.cc:2369] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:26:19.380874 17351 tablet.cc:2369] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:26:19.555401 17414 raft_consensus.cc:482] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:26:19.555738 17414 raft_consensus.cc:504] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2267e21c5f34498a58d65caeeb39392" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37521 } } peers { permanent_uuid: "9d6864eee349464fa54c8421f889970f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 45971 } } peers { permanent_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40277 } }
I20260806 12:26:19.557626 17414 leader_election.cc:288] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2267e21c5f34498a58d65caeeb39392 (127.28.210.66:37521), f741699fcac84ffeb78dbf0ac6cc6bd7 (127.28.210.67:40277)
I20260806 12:26:19.558874 17412 raft_consensus.cc:482] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:26:19.559063 17412 raft_consensus.cc:504] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2267e21c5f34498a58d65caeeb39392" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37521 } } peers { permanent_uuid: "9d6864eee349464fa54c8421f889970f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 45971 } } peers { permanent_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40277 } }
I20260806 12:26:19.560482 17412 leader_election.cc:288] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2267e21c5f34498a58d65caeeb39392 (127.28.210.66:37521), 9d6864eee349464fa54c8421f889970f (127.28.210.65:45971)
I20260806 12:26:19.582924 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.561026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:37521 (local address 127.28.210.67:33707)
0806 12:26:19.567542 (+ 6516us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:19.567557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:19.567675 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:19.568125 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:19.568128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:19.568146 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:19.568281 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.568287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.569317 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.569320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.580478 (+ 11158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.580493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.582135 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.582139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.582327 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.582354 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 12:26:19.582422 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 12:26:19.582519 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6432,"thread_start_us":136,"threads_started":1}
I20260806 12:26:19.583530 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.562799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:33707 (local address 127.28.210.66:37521)
0806 12:26:19.564781 (+ 1982us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:19.564790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:19.567760 (+ 2970us) server_negotiation.cc:407] Connection header received
0806 12:26:19.567812 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:19.567815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:19.567861 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:19.567950 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:19.568410 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.569142 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.580661 (+ 11519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.581865 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.582007 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.582060 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:19.583225 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0806 12:26:19.583330 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1896,"thread_start_us":245,"threads_started":1}
I20260806 12:26:19.584247 17044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7405bf34b0e641b2ae0fab2b64951d45" candidate_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2267e21c5f34498a58d65caeeb39392" is_pre_election: true
I20260806 12:26:19.584627 17044 raft_consensus.cc:2454] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741699fcac84ffeb78dbf0ac6cc6bd7 in term 0.
I20260806 12:26:19.585278 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.562350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:35669 (local address 127.28.210.65:45971)
0806 12:26:19.562506 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:19.562511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:19.564577 (+ 2066us) server_negotiation.cc:407] Connection header received
0806 12:26:19.566265 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:19.566269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:19.566326 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:19.566374 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:19.568460 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.569261 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.582567 (+ 13306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.583136 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.583240 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.583293 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:19.585044 (+ 1751us) server_negotiation.cc:299] Negotiation successful
0806 12:26:19.585157 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 12:26:19.586146 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.560795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:45971 (local address 127.28.210.67:35669)
0806 12:26:19.564469 (+ 3674us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:19.564487 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:19.565369 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:19.566511 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:19.566515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:19.566538 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:19.566767 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.566778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.571056 (+ 4278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.571060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.572504 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.572520 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.583791 (+ 11271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.583794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.583964 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.583992 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 12:26:19.585846 (+ 1854us) client_negotiation.cc:240] Negotiation successful
0806 12:26:19.585976 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3588,"thread_start_us":125,"threads_started":1}
I20260806 12:26:19.586500 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.558099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:37521 (local address 127.28.210.65:59683)
0806 12:26:19.564988 (+ 6889us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:19.565003 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:19.565191 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:19.565808 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:19.565811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:19.565833 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:19.566097 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.566109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.569415 (+ 3306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.569418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.577658 (+ 8240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.577670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.582195 (+ 4525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.582200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.582384 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.582413 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 12:26:19.584289 (+ 1876us) client_negotiation.cc:240] Negotiation successful
0806 12:26:19.584479 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6805,"thread_start_us":170,"threads_started":1}
I20260806 12:26:19.588023 16849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7405bf34b0e641b2ae0fab2b64951d45" candidate_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6864eee349464fa54c8421f889970f" is_pre_election: true
I20260806 12:26:19.588346 16849 raft_consensus.cc:2454] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f741699fcac84ffeb78dbf0ac6cc6bd7 in term 0.
I20260806 12:26:19.588986 17179 leader_election.cc:302] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d6864eee349464fa54c8421f889970f, f741699fcac84ffeb78dbf0ac6cc6bd7; no voters:
I20260806 12:26:19.589360 17412 raft_consensus.cc:2781] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:26:19.589445 17412 raft_consensus.cc:482] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:26:19.589486 17412 raft_consensus.cc:3037] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:19.591570 17412 raft_consensus.cc:504] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2267e21c5f34498a58d65caeeb39392" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37521 } } peers { permanent_uuid: "9d6864eee349464fa54c8421f889970f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 45971 } } peers { permanent_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40277 } }
I20260806 12:26:19.592607 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.558869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:59683 (local address 127.28.210.66:37521)
0806 12:26:19.559029 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:19.559034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:19.565089 (+ 6055us) server_negotiation.cc:407] Connection header received
0806 12:26:19.565531 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:19.565536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:19.565593 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:19.565645 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:19.567480 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.568292 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.580004 (+ 11712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.580720 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.580856 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.580919 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:19.584166 (+ 3247us) server_negotiation.cc:299] Negotiation successful
0806 12:26:19.584252 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 12:26:19.593190 17047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7405bf34b0e641b2ae0fab2b64951d45" candidate_uuid: "9d6864eee349464fa54c8421f889970f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2267e21c5f34498a58d65caeeb39392" is_pre_election: true
I20260806 12:26:19.593297 16849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7405bf34b0e641b2ae0fab2b64951d45" candidate_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6864eee349464fa54c8421f889970f"
I20260806 12:26:19.593356 17047 raft_consensus.cc:2454] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d6864eee349464fa54c8421f889970f in term 0.
I20260806 12:26:19.593462 16849 raft_consensus.cc:3037] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:19.593703 17044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7405bf34b0e641b2ae0fab2b64951d45" candidate_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2267e21c5f34498a58d65caeeb39392"
I20260806 12:26:19.593811 17044 raft_consensus.cc:3037] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:19.594972 16849 raft_consensus.cc:2454] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741699fcac84ffeb78dbf0ac6cc6bd7 in term 1.
I20260806 12:26:19.595387 17044 raft_consensus.cc:2454] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f741699fcac84ffeb78dbf0ac6cc6bd7 in term 1.
I20260806 12:26:19.595335 16777 leader_election.cc:302] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d6864eee349464fa54c8421f889970f, e2267e21c5f34498a58d65caeeb39392; no voters:
I20260806 12:26:19.595636 17414 raft_consensus.cc:2741] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260806 12:26:19.592059 17412 leader_election.cc:288] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [CANDIDATE]: Term 1 election: Requested vote from peers e2267e21c5f34498a58d65caeeb39392 (127.28.210.66:37521), 9d6864eee349464fa54c8421f889970f (127.28.210.65:45971)
I20260806 12:26:19.595389 17179 leader_election.cc:302] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d6864eee349464fa54c8421f889970f, f741699fcac84ffeb78dbf0ac6cc6bd7; no voters:
I20260806 12:26:19.596329 17412 raft_consensus.cc:2781] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:26:19.596650 17412 raft_consensus.cc:686] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 1 LEADER]: Becoming Leader. State: Replica: f741699fcac84ffeb78dbf0ac6cc6bd7, State: Running, Role: LEADER
I20260806 12:26:19.597060 17412 consensus_queue.cc:237] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2267e21c5f34498a58d65caeeb39392" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37521 } } peers { permanent_uuid: "9d6864eee349464fa54c8421f889970f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 45971 } } peers { permanent_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40277 } }
I20260806 12:26:19.600617 16646 catalog_manager.cc:5495] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f741699fcac84ffeb78dbf0ac6cc6bd7 (127.28.210.67). New cstate: current_term: 1 leader_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2267e21c5f34498a58d65caeeb39392" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d6864eee349464fa54c8421f889970f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 45971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40277 } health_report { overall_health: HEALTHY } } }
I20260806 12:26:19.605579 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.562600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:37489 (local address 127.28.210.67:40277)
0806 12:26:19.562733 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:19.562737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:19.567387 (+ 4650us) server_negotiation.cc:407] Connection header received
0806 12:26:19.576460 (+ 9073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:19.576464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:19.576526 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:19.576579 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:19.577824 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.578580 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.603999 (+ 25419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.604671 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.604818 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.604879 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:19.605325 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 12:26:19.605433 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 12:26:19.606257 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.562039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:40277 (local address 127.28.210.65:37489)
0806 12:26:19.567243 (+ 5204us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:19.567262 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:19.576240 (+ 8978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:19.576748 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:19.576751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:19.576776 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:19.577237 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.577249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.578734 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.578738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.603692 (+ 24954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:19.603859 (+ 167us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.604987 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.604992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:19.605138 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.605172 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 12:26:19.605978 (+ 806us) client_negotiation.cc:240] Negotiation successful
0806 12:26:19.606092 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5113,"thread_start_us":152,"threads_started":1}
I20260806 12:26:19.607302 17252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7405bf34b0e641b2ae0fab2b64951d45" candidate_uuid: "9d6864eee349464fa54c8421f889970f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f741699fcac84ffeb78dbf0ac6cc6bd7" is_pre_election: true
I20260806 12:26:19.660418 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.656626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.28.210.67:40277)
0806 12:26:19.656740 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:19.656745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:19.656775 (+ 30us) server_negotiation.cc:407] Connection header received
0806 12:26:19.656975 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:19.656978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:19.657039 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:19.657131 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:26:19.657365 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.657872 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.658743 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:19.658978 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:19.659070 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:19.659448 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:26:19.659941 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:26:19.660056 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:19.660150 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 12:26:19.660211 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 12:26:19.667578 17044 raft_consensus.cc:1261] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392 [term 1 FOLLOWER]: Refusing update from remote peer f741699fcac84ffeb78dbf0ac6cc6bd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:26:19.667578 16849 raft_consensus.cc:1261] T 7405bf34b0e641b2ae0fab2b64951d45 P 9d6864eee349464fa54c8421f889970f [term 1 FOLLOWER]: Refusing update from remote peer f741699fcac84ffeb78dbf0ac6cc6bd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:26:19.668345 17412 consensus_queue.cc:1035] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2267e21c5f34498a58d65caeeb39392" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:26:19.668602 17434 consensus_queue.cc:1035] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d6864eee349464fa54c8421f889970f" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 45971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:26:19.681742 17443 mvcc.cc:205] Tried to move back new op lower bound from 7315534559910502400 to 7315534559632449536. Current Snapshot: MvccSnapshot[applied={T|T < 7315534559910502400}]
I20260806 12:26:19.683756 17444 mvcc.cc:205] Tried to move back new op lower bound from 7315534559910502400 to 7315534559632449536. Current Snapshot: MvccSnapshot[applied={T|T < 7315534559910502400}]
I20260806 12:26:19.687479 17446 mvcc.cc:205] Tried to move back new op lower bound from 7315534559910502400 to 7315534559632449536. Current Snapshot: MvccSnapshot[applied={T|T < 7315534559910502400}]
I20260806 12:26:19.698809 16646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:26:19.699019 16646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:26:19.701867 16646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c02f2bbb889945a988e53ec6c13da4dd: Sending DeleteTablet for 3 replicas of tablet 7405bf34b0e641b2ae0fab2b64951d45
I20260806 12:26:19.702792 17043 tablet_service.cc:1514] Processing DeleteTablet for tablet 7405bf34b0e641b2ae0fab2b64951d45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:26:19 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:47744
I20260806 12:26:19.702966 16823 tablet_service.cc:1514] Processing DeleteTablet for tablet 7405bf34b0e641b2ae0fab2b64951d45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:26:19 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:37950
I20260806 12:26:19.703096 17223 tablet_service.cc:1514] Processing DeleteTablet for tablet 7405bf34b0e641b2ae0fab2b64951d45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:26:19 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:57700
I20260806 12:26:19.704442 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 16700
I20260806 12:26:19.712459 17465 tablet_replica.cc:330] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7: stopping tablet replica
I20260806 12:26:19.712950 17465 raft_consensus.cc:2229] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:26:19.713483 17465 raft_consensus.cc:2258] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:26:19.714051 17462 tablet_replica.cc:330] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392: stopping tablet replica
I20260806 12:26:19.714422 17462 raft_consensus.cc:2229] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:26:19.714767 17462 raft_consensus.cc:2258] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:26:19.716506 17462 ts_tablet_manager.cc:1905] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:26:19.717092 17465 ts_tablet_manager.cc:1905] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:26:19.720247 17465 ts_tablet_manager.cc:1918] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:26:19.720454 17465 log.cc:1196] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/7405bf34b0e641b2ae0fab2b64951d45
I20260806 12:26:19.720826 17465 ts_tablet_manager.cc:1939] T 7405bf34b0e641b2ae0fab2b64951d45 P f741699fcac84ffeb78dbf0ac6cc6bd7: Deleting consensus metadata
I20260806 12:26:19.722307 16623 catalog_manager.cc:4841] TS f741699fcac84ffeb78dbf0ac6cc6bd7 (127.28.210.67:40277): tablet 7405bf34b0e641b2ae0fab2b64951d45 (table test-table [id=18b3ae8771ff44ba8fac7f1e44538665]) successfully deleted
W20260806 12:26:19.727537 16623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:45971 (error 108)
I20260806 12:26:19.729214 17462 ts_tablet_manager.cc:1918] T 7405bf34b0e641b2ae0fab2b64951d45 P e2267e21c5f34498a58d65caeeb39392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:26:19.729240 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 16948
I20260806 12:26:19.740491 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.728898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:45971 (local address 127.0.0.1:60360)
0806 12:26:19.740250 (+ 11352us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:19.740352 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11249,"thread_start_us":59,"threads_started":1}
W20260806 12:26:19.740788 16623 catalog_manager.cc:4584] TS 9d6864eee349464fa54c8421f889970f (127.28.210.65:45971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7405bf34b0e641b2ae0fab2b64951d45: Network error: Client connection negotiation failed: client connection to 127.28.210.65:45971: connect: Connection refused (error 111)
I20260806 12:26:19.748610 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 17145
I20260806 12:26:19.748848 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:19.748436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:37521 (local address 127.0.0.1:37694)
0806 12:26:19.748678 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:19.748735 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.66:37521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260806 12:26:19.763221 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 16596
I20260806 12:26:19.777968 29513 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 12:26:20.123569 29513 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 12:26:20.123698 29513 mini_ranger.cc:66] Stopping Ranger...
I20260806 12:26:20.469179 29513 mini_ranger.cc:68] Stopped Ranger
2026-08-06 12:26:20.469 UTC [15181] LOG: received smart shutdown request
2026-08-06 12:26:20.474 UTC [15181] LOG: background worker "logical replication launcher" (PID 15195) exited with exit code 1
2026-08-06 12:26:20.475 UTC [15187] LOG: shutting down
2026-08-06 12:26:20.659 UTC [15181] LOG: database system is shut down
2026-08-06T12:26:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53115 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:26:20 dist-test-slave-lgr5 krb5kdc[17482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:26:20 dist-test-slave-lgr5 krb5kdc[17482](info): set up 2 sockets
Aug 06 12:26:20 dist-test-slave-lgr5 krb5kdc[17482](info): commencing operation
krb5kdc: starting...
W20260806 12:26:21.906653 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:26:21 dist-test-slave-lgr5 krb5kdc[17482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:26:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:26:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:26:22.043913 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:35717
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:46443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:35717,127.28.210.125:36685,127.28.210.124:33415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:22.157120 17499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:22.157416 17499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:22.157475 17499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:22.161908 17499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:26:22.161995 17499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:22.162025 17499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:22.162048 17499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:26:22.162079 17499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:26:22.167943 17499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:46443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:35717,127.28.210.125:36685,127.28.210.124:33415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:22.169216 17499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:22.170297 17499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:22.179298 17506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:22.179298 17509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:22.179298 17507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:22.179347 17499 server_base.cc:1034] running on GCE node
I20260806 12:26:22.180255 17499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:22.181502 17499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:22.182674 17499 hybrid_clock.cc:648] HybridClock initialized: now 1786019182182655 us; error 38 us; skew 500 ppm
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019182, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:22.185544 17499 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:26:22.186532 17499 webserver.cc:460] Webserver started at http://127.28.210.126:43957/ using document root <none> and password file <none>
I20260806 12:26:22.187094 17499 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:22.187160 17499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:22.187366 17499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:26:22.189512 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "94e73341d90c4b8d922eac0f79d420ca"
format_stamp: "Formatted at 2026-08-06 12:26:22 on dist-test-slave-lgr5"
server_key: "34cfa8aa12b11061b22995b977f1c899"
server_key_iv: "940f4b2b7d62cb47eddccc3a664f6524"
server_key_version: "encryptionkey@0"
I20260806 12:26:22.190092 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "94e73341d90c4b8d922eac0f79d420ca"
format_stamp: "Formatted at 2026-08-06 12:26:22 on dist-test-slave-lgr5"
server_key: "34cfa8aa12b11061b22995b977f1c899"
server_key_iv: "940f4b2b7d62cb47eddccc3a664f6524"
server_key_version: "encryptionkey@0"
I20260806 12:26:22.193876 17499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 12:26:22.196416 17516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:22.197686 17499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 12:26:22.197821 17499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "94e73341d90c4b8d922eac0f79d420ca"
format_stamp: "Formatted at 2026-08-06 12:26:22 on dist-test-slave-lgr5"
server_key: "34cfa8aa12b11061b22995b977f1c899"
server_key_iv: "940f4b2b7d62cb47eddccc3a664f6524"
server_key_version: "encryptionkey@0"
I20260806 12:26:22.197928 17499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:22.214416 17499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:22.217641 17499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:22.217839 17499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:22.225235 17499 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:35717
I20260806 12:26:22.225266 17597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:35717 every 8 connection(s)
I20260806 12:26:22.226348 17499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:26:22.229737 17598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:22.231498 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 17499
I20260806 12:26:22.231652 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:26:22.232020 29513 external_mini_cluster.cc:1428] Setting key 1ee58280389b3a4b9803bf935ddbe2b3
I20260806 12:26:22.234776 17598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.210.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.125@KRBTEST.COM" created.
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019182, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:22.251088 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.237336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.28.210.126:35717)
0806 12:26:22.237935 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.237958 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.238248 (+ 290us) server_negotiation.cc:407] Connection header received
0806 12:26:22.239377 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.239383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.239643 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.239863 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.240751 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.241891 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.243012 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.243440 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.243573 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.246211 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.246219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.246233 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.246261 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.247900 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.248462 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.248467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.248468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.248579 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.248902 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.248904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.248905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.249302 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.249460 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.249973 (+ 513us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.250216 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":408,"thread_start_us":177,"threads_started":1}
I20260806 12:26:22.251092 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.237336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35717 (local address 127.0.0.1:58804)
0806 12:26:22.238063 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.238117 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:22.239021 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:22.239901 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:22.239906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:22.240292 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:22.240579 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.240594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.242032 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.242035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.242888 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.242899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.243640 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.243650 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.243753 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.244355 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:22.244373 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:22.246035 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:22.248024 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.248033 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.248044 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.248350 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.248668 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.248671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.248672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.248817 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.249462 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:22.249470 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:22.249671 (+ 201us) client_negotiation.cc:769] Sending connection context
0806 12:26:22.249857 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 12:26:22.250313 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":95,"spinlock_wait_cycles":57600,"thread_start_us":177,"threads_started":1}
I20260806 12:26:22.253750 17598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:26:22.254554 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.254230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:36685 (local address 127.0.0.1:38886)
0806 12:26:22.254338 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.254445 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 12:26:22.254724 17523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.210.125:36685: connect: Connection refused (error 111)
I20260806 12:26:22.255614 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.255414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:36685 (local address 127.0.0.1:38902)
0806 12:26:22.255515 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.255559 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 12:26:22.255735 17598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.125:36685: Network error: Client connection negotiation failed: client connection to 127.28.210.125:36685: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.125.keytab.
Entry for principal kudu/127.28.210.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.125.keytab.
I20260806 12:26:22.262652 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.125:36685
--webserver_interface=127.28.210.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.125.keytab
--principal=kudu/127.28.210.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:46443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:35717,127.28.210.125:36685,127.28.210.124:33415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:26:22.304999 17598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } attempt: 1
I20260806 12:26:22.305836 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.305543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:36685 (local address 127.0.0.1:38908)
0806 12:26:22.305690 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.305755 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 12:26:22.306725 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.306460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:36685 (local address 127.0.0.1:38920)
0806 12:26:22.306611 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.306661 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260806 12:26:22.306885 17598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.125:36685: Network error: Client connection negotiation failed: client connection to 127.28.210.125:36685: connect: Connection refused (error 111)
I20260806 12:26:22.375064 17598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } attempt: 2
I20260806 12:26:22.376047 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.375690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:36685 (local address 127.0.0.1:38924)
0806 12:26:22.375889 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.375963 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 12:26:22.376820 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.376590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:36685 (local address 127.0.0.1:38938)
0806 12:26:22.376700 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.376754 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 12:26:22.376941 17598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.125:36685: Network error: Client connection negotiation failed: client connection to 127.28.210.125:36685: connect: Connection refused (error 111)
W20260806 12:26:22.378350 17609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:22.378609 17609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:22.378677 17609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:22.382773 17609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:26:22.382838 17609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:22.382864 17609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:22.382887 17609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:26:22.382918 17609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:26:22.388469 17609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:46443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:35717,127.28.210.125:36685,127.28.210.124:33415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.125.keytab
--principal=kudu/127.28.210.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.125:36685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.210.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:22.389719 17609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:22.390777 17609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:22.399801 17616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:22.399801 17619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:22.399801 17624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:22.400208 17609 server_base.cc:1034] running on GCE node
I20260806 12:26:22.401013 17609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:22.402465 17609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:22.403674 17609 hybrid_clock.cc:648] HybridClock initialized: now 1786019182403651 us; error 42 us; skew 500 ppm
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019182, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:22.407027 17609 init.cc:377] Logged in from keytab as kudu/127.28.210.125@KRBTEST.COM (short username kudu)
I20260806 12:26:22.408070 17609 webserver.cc:460] Webserver started at http://127.28.210.125:36563/ using document root <none> and password file <none>
I20260806 12:26:22.408730 17609 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:22.408804 17609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:22.409027 17609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:26:22.411273 17609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/data/instance:
uuid: "6296f00d82044e76b2999e98a5b4c1dd"
format_stamp: "Formatted at 2026-08-06 12:26:22 on dist-test-slave-lgr5"
server_key: "191bb2589e36c01b422c4dd92f5114e1"
server_key_iv: "5da479a40ca56e131633fb55a72b9aba"
server_key_version: "encryptionkey@0"
I20260806 12:26:22.411935 17609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/wal/instance:
uuid: "6296f00d82044e76b2999e98a5b4c1dd"
format_stamp: "Formatted at 2026-08-06 12:26:22 on dist-test-slave-lgr5"
server_key: "191bb2589e36c01b422c4dd92f5114e1"
server_key_iv: "5da479a40ca56e131633fb55a72b9aba"
server_key_version: "encryptionkey@0"
I20260806 12:26:22.415915 17609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 12:26:22.418498 17634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:22.419792 17609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 12:26:22.419919 17609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/wal
uuid: "6296f00d82044e76b2999e98a5b4c1dd"
format_stamp: "Formatted at 2026-08-06 12:26:22 on dist-test-slave-lgr5"
server_key: "191bb2589e36c01b422c4dd92f5114e1"
server_key_iv: "5da479a40ca56e131633fb55a72b9aba"
server_key_version: "encryptionkey@0"
I20260806 12:26:22.420025 17609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:22.440697 17609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:22.443629 17609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:22.443807 17609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:22.450735 17609 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.125:36685
I20260806 12:26:22.451006 17713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.125:36685 every 8 connection(s)
I20260806 12:26:22.451706 17609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
I20260806 12:26:22.454556 17714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:22.458489 17714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:26:22.460628 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 17609
I20260806 12:26:22.460752 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-1/wal/instance
I20260806 12:26:22.461020 29513 external_mini_cluster.cc:1428] Setting key 33319872b41cea31680667f3057b3ecb
I20260806 12:26:22.468118 17598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } attempt: 3
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019182, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.125@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.124@KRBTEST.COM" created.
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019182, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.125@KRBTEST.COM
I20260806 12:26:22.475598 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.460437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.28.210.126:35717)
0806 12:26:22.460690 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.460695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.461818 (+ 1123us) server_negotiation.cc:407] Connection header received
0806 12:26:22.463015 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.463019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.463085 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.463190 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.464523 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.465266 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.466686 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.467048 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.467176 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.470238 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.470242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.470245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.470276 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.472070 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.472705 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.472710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.472712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.472828 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.473221 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.473225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.473227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.473496 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.473607 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.475265 (+ 1658us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.475415 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"spinlock_wait_cycles":7936}
I20260806 12:26:22.476351 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.460979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35717 (local address 127.0.0.1:58808)
0806 12:26:22.461623 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.461683 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:22.462744 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:22.463367 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:22.463374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:22.463733 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:22.464320 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.464336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.465429 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.465432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.466493 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.466506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.467261 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.467265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.467454 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.468190 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:22.468210 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:22.470073 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:22.472224 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.472232 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.472252 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.472589 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.472941 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.472945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.472947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.473117 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.473742 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:22.473751 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:22.474082 (+ 331us) client_negotiation.cc:769] Sending connection context
0806 12:26:22.474356 (+ 274us) client_negotiation.cc:240] Negotiation successful
0806 12:26:22.474771 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":150,"threads_started":1}
I20260806 12:26:22.478250 17714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:26:22.480391 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.468700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:36685 (local address 127.0.0.1:38954)
0806 12:26:22.468936 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.468951 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:22.469034 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:22.469981 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:22.469984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:22.470166 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:22.470298 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.470306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.471808 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.471812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.472399 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.472411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.473041 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.473044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.473156 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.473685 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:22.473699 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:22.475276 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:22.478142 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.478146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.478148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.478421 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.478796 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.478800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.478802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.478930 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.479770 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:22.479773 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:22.479882 (+ 109us) client_negotiation.cc:769] Sending connection context
0806 12:26:22.480004 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 12:26:22.480210 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"spinlock_wait_cycles":3840}
I20260806 12:26:22.480541 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.468806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.28.210.125:36685)
0806 12:26:22.469440 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.469453 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.469481 (+ 28us) server_negotiation.cc:407] Connection header received
0806 12:26:22.469529 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.469534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.469782 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.469967 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.470436 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.471682 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.472518 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.472916 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.473017 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.475464 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.475489 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.475507 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.475538 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.477992 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.478582 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.478587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.478589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.478676 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.479345 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.479348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.479349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.479661 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.479850 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.480143 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.480316 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":123,"threads_started":1}
I20260806 12:26:22.482685 17598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:26:22.483656 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.483324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49640)
0806 12:26:22.483484 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.483555 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 12:26:22.485037 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.484668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49644)
0806 12:26:22.484886 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.484936 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260806 12:26:22.485217 17598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:33415: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019182, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.125@KRBTEST.COM for kudu/127.28.210.125@KRBTEST.COM
I20260806 12:26:22.489228 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.478831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:36685 (local address 127.0.0.1:38968)
0806 12:26:22.478983 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.478997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:22.479068 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:22.479670 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:22.479673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:22.479875 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:22.479994 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.480002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.481538 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.481542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.482386 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.482400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.483494 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.483498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.483597 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.484179 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:22.484190 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:22.485724 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:22.487665 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.487669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.487670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.487913 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.488238 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.488240 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.488242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.488325 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.488759 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:22.488762 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:22.488857 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 12:26:22.488947 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 12:26:22.489095 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 12:26:22.489399 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.479101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.28.210.125:36685)
0806 12:26:22.479442 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.479449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.479461 (+ 12us) server_negotiation.cc:407] Connection header received
0806 12:26:22.479501 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.479503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.479542 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.479619 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.480167 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.481407 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.482538 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.483353 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.483857 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.485926 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.485930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.485933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.485967 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.487547 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.488020 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.488023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.488025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.488144 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.488432 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.488435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.488436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.488675 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.488761 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.488997 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.489213 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":276,"thread_start_us":155,"threads_started":1}
I20260806 12:26:22.490173 17714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:26:22.490972 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.490698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49656)
0806 12:26:22.490815 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.490905 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 12:26:22.491201 17635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
I20260806 12:26:22.492198 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.491937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49658)
0806 12:26:22.492036 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.492134 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260806 12:26:22.492336 17714 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:33415: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.124.keytab.
Entry for principal kudu/127.28.210.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.124.keytab.
I20260806 12:26:22.495343 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.124:33415
--webserver_interface=127.28.210.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.124.keytab
--principal=kudu/127.28.210.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:46443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:35717,127.28.210.125:36685,127.28.210.124:33415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:26:22.516379 17598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } attempt: 1
I20260806 12:26:22.517383 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.517000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49660)
0806 12:26:22.517194 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.517260 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 12:26:22.518101 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.517893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49666)
0806 12:26:22.518000 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.518045 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 12:26:22.518249 17598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:33415: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
I20260806 12:26:22.541600 17714 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } attempt: 1
I20260806 12:26:22.542605 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.542223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49678)
0806 12:26:22.542407 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.542487 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 12:26:22.543347 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.543125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49692)
0806 12:26:22.543225 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.543279 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 12:26:22.543536 17714 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:33415: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
I20260806 12:26:22.593428 17598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } attempt: 2
I20260806 12:26:22.594321 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.593997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49700)
0806 12:26:22.594157 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.594223 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 12:26:22.595196 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.594943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49710)
0806 12:26:22.595074 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.595124 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 12:26:22.595373 17598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:33415: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
I20260806 12:26:22.611711 17714 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } attempt: 2
I20260806 12:26:22.612650 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.612309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49720)
0806 12:26:22.612480 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.612554 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 12:26:22.613485 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.613231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49722)
0806 12:26:22.613343 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.613401 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 12:26:22.613665 17714 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:33415: Network error: Client connection negotiation failed: client connection to 127.28.210.124:33415: connect: Connection refused (error 111)
W20260806 12:26:22.616391 17724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:22.616665 17724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:22.616714 17724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:22.620824 17724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:26:22.620888 17724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:22.620908 17724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:22.620923 17724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:26:22.620941 17724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:26:22.626643 17724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:46443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:35717,127.28.210.125:36685,127.28.210.124:33415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.124.keytab
--principal=kudu/127.28.210.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.124:33415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.210.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:22.627880 17724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:22.629016 17724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:22.637825 17734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:22.637827 17736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:22.637827 17741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:22.637948 17724 server_base.cc:1034] running on GCE node
I20260806 12:26:22.638733 17724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:22.639904 17724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:22.641119 17724 hybrid_clock.cc:648] HybridClock initialized: now 1786019182641096 us; error 56 us; skew 500 ppm
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019182, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:22.644022 17724 init.cc:377] Logged in from keytab as kudu/127.28.210.124@KRBTEST.COM (short username kudu)
I20260806 12:26:22.645015 17724 webserver.cc:460] Webserver started at http://127.28.210.124:36943/ using document root <none> and password file <none>
I20260806 12:26:22.645583 17724 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:22.645642 17724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:22.645830 17724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:26:22.648027 17724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/data/instance:
uuid: "e44df286eac6439eb4abf8c015dd81a9"
format_stamp: "Formatted at 2026-08-06 12:26:22 on dist-test-slave-lgr5"
server_key: "99f385895b79e151aab8e936ba9afda9"
server_key_iv: "2611160ca39a4a3948e91e810c420da6"
server_key_version: "encryptionkey@0"
I20260806 12:26:22.648767 17724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/wal/instance:
uuid: "e44df286eac6439eb4abf8c015dd81a9"
format_stamp: "Formatted at 2026-08-06 12:26:22 on dist-test-slave-lgr5"
server_key: "99f385895b79e151aab8e936ba9afda9"
server_key_iv: "2611160ca39a4a3948e91e810c420da6"
server_key_version: "encryptionkey@0"
I20260806 12:26:22.652930 17724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:26:22.655493 17750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:22.656849 17724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:22.656970 17724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/wal
uuid: "e44df286eac6439eb4abf8c015dd81a9"
format_stamp: "Formatted at 2026-08-06 12:26:22 on dist-test-slave-lgr5"
server_key: "99f385895b79e151aab8e936ba9afda9"
server_key_iv: "2611160ca39a4a3948e91e810c420da6"
server_key_version: "encryptionkey@0"
I20260806 12:26:22.657068 17724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:22.682507 17724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:22.685519 17724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:22.685714 17724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:22.692674 17724 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.124:33415
I20260806 12:26:22.692999 17832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.124:33415 every 8 connection(s)
I20260806 12:26:22.693722 17724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
I20260806 12:26:22.694537 17598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } attempt: 3
I20260806 12:26:22.696772 17833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:22.700598 17833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019182, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.124@KRBTEST.COM
I20260806 12:26:22.703231 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 17724
I20260806 12:26:22.703370 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786018995816160-29513-0/minicluster-data/master-2/wal/instance
I20260806 12:26:22.703711 29513 external_mini_cluster.cc:1428] Setting key b3d9afa37153cb7b8092c31c90b0d783
I20260806 12:26:22.704844 17714 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } attempt: 3
I20260806 12:26:22.707468 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.695035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49724)
0806 12:26:22.695187 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.695205 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:22.695307 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:22.697594 (+ 2287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:22.697598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:22.697798 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:22.697915 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.697924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.699156 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.699159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.699673 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.699682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.700493 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.700497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.700599 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.701132 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:22.701147 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:22.702612 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:22.705317 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.705321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.705323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.705557 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.705954 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.705958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.705960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.706060 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.706920 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:22.706923 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:22.707026 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 12:26:22.707131 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 12:26:22.707267 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 12:26:22.709121 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.695799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.28.210.124:33415)
0806 12:26:22.696256 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.696274 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.696311 (+ 37us) server_negotiation.cc:407] Connection header received
0806 12:26:22.696990 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.697007 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.697325 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.697629 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.698070 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.699029 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.699782 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.700352 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.700498 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.702842 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.702847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.702860 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.702888 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.705186 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.705698 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.705704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.705710 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.705847 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.706167 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.706171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.706173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.706744 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.706858 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.707779 (+ 921us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.708122 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":129,"threads_started":1}
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019182, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.124@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019182, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.125@KRBTEST.COM for kudu/127.28.210.124@KRBTEST.COM
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.124@KRBTEST.COM
I20260806 12:26:22.713409 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.702596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.28.210.126:35717)
0806 12:26:22.702720 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.702724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.703242 (+ 518us) server_negotiation.cc:407] Connection header received
0806 12:26:22.703400 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.703403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.703493 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.703579 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.704596 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.705268 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.706354 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.706664 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.706755 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.710028 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.710032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.710035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.710063 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.711502 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.712001 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.712004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.712006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.712109 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.712414 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.712417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.712418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.712645 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.712736 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.713148 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.713252 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 12:26:22.713430 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.702647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35717 (local address 127.0.0.1:58810)
0806 12:26:22.703097 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.703149 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:22.703274 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:22.703711 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:22.703718 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:22.704269 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:22.704448 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.704457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.705412 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.705415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.706249 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.706260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.706870 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.706874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.707004 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.707500 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:22.707514 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:22.709871 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:22.711665 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.711673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.711677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.711875 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.712215 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.712218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.712220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.712311 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.712791 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:22.712798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:22.712937 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 12:26:22.713073 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 12:26:22.713252 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":158,"threads_started":1}
I20260806 12:26:22.714358 17598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca: Bootstrap starting.
I20260806 12:26:22.714707 17833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:26:22.717633 17598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:22.717672 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.705305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49744)
0806 12:26:22.705681 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.705696 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:22.705776 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:22.706255 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:22.706259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:22.706453 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:22.706556 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.706563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.708069 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.708073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.708883 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.708896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.709820 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.709824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.709907 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.710662 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:22.710678 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:22.712384 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:22.716000 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.716003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.716006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.716291 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.716646 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.716650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.716651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.716733 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.717186 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:22.717189 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:22.717286 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 12:26:22.717367 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 12:26:22.717516 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285}
I20260806 12:26:22.718205 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.704466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.28.210.124:33415)
0806 12:26:22.705738 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.705747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.706708 (+ 961us) server_negotiation.cc:407] Connection header received
0806 12:26:22.706744 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.706747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.706785 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.706859 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.707771 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.708523 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.709576 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.709919 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.710011 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.713270 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.713273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.713275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.713300 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.715950 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.716405 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.716409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.716411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.716497 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.716791 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.716794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.716795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.717071 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.717245 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.717928 (+ 683us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.718042 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":1205,"spinlock_wait_cycles":7296,"thread_start_us":114,"threads_started":1}
I20260806 12:26:22.718562 17598 log.cc:824] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:22.718674 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.705412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.28.210.124:33415)
0806 12:26:22.706050 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.706058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.706070 (+ 12us) server_negotiation.cc:407] Connection header received
0806 12:26:22.706100 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.706102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.706145 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.706219 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.706686 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.707929 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.709052 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.709681 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.710610 (+ 929us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.712606 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.712610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.712612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.712633 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.715859 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.716462 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.716465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.716467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.716547 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.716833 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.716835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.716837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.717071 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.718121 (+ 1050us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.718191 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.718554 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":577,"spinlock_wait_cycles":2048,"thread_start_us":119,"threads_started":1}
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019182, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.124@KRBTEST.COM for kudu/127.28.210.125@KRBTEST.COM
I20260806 12:26:22.720945 17598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca: No bootstrap required, opened a new log
I20260806 12:26:22.722476 17714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd: Bootstrap starting.
I20260806 12:26:22.724814 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.715482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.28.210.125:36685)
0806 12:26:22.715592 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.715596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.715609 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:26:22.715699 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.715702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.715742 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.715788 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.716347 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.717086 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.718083 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.718416 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.718507 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.721243 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.721247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.721250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.721275 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.722784 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.723324 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.723328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.723330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.723419 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.723774 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.723777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.723779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.724064 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.724166 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.724540 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.724662 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 12:26:22.724833 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.715249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:36685 (local address 127.0.0.1:38970)
0806 12:26:22.715514 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.715527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:22.715599 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:22.715928 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:22.715931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:22.716126 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:22.716214 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.716221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.717223 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.717226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.717948 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.717960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.719135 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.719139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.719218 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.719685 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:22.719699 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:22.721106 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:22.722904 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.722908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.722910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.723156 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.723516 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.723519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.723521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.723666 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.724241 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:22.724244 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:22.724343 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 12:26:22.724476 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 12:26:22.724611 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260806 12:26:22.724712 17598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } }
I20260806 12:26:22.724953 17598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:22.725003 17598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e73341d90c4b8d922eac0f79d420ca, State: Initialized, Role: FOLLOWER
I20260806 12:26:22.725441 17598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } }
I20260806 12:26:22.725905 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } } }
I20260806 12:26:22.726035 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:26:22.726703 17714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:22.726780 17598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:26:22.727209 17833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:26:22.728219 17714 log.cc:824] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd: Log is configured to *not* fsync() on all Append() calls
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019182, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.124@KRBTEST.COM for kudu/127.28.210.124@KRBTEST.COM
W20260806 12:26:22.733156 17860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:26:22.733286 17860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:26:22.734864 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.719858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.28.210.126:35717)
0806 12:26:22.723430 (+ 3572us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.723434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.723447 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:26:22.723485 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.723488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.723531 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.723597 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.724467 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.725191 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.726382 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.727180 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.727279 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.730359 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.730363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.730366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.730392 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.732530 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.733043 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.733047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.733049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.733138 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.733419 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.733423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.733425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.733659 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.733852 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.734550 (+ 698us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.734688 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3477}
I20260806 12:26:22.736768 17714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd: No bootstrap required, opened a new log
I20260806 12:26:22.732682 17499 master_runner.cc:186] Error getting master registration for 127.28.210.125:36685: OK, instance_id { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" instance_seqno: 1786019182448838 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 12:26:22.736960 17499 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 12:26:22.740552 17714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } }
I20260806 12:26:22.740927 17714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:22.740995 17714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6296f00d82044e76b2999e98a5b4c1dd, State: Initialized, Role: FOLLOWER
I20260806 12:26:22.741676 17714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } }
I20260806 12:26:22.741911 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.727735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:33415 (local address 127.0.0.1:49754)
0806 12:26:22.727955 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:22.727969 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:22.728049 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:22.728538 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:22.728541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:22.728717 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:22.728825 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.728833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.729776 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.729779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.730488 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:22.730501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.731057 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.731060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:22.731129 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.731591 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:22.731605 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:22.733111 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:22.734820 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.734824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.734826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.735085 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.740713 (+ 5628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:22.740717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:22.740720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:22.740839 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:22.741359 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:22.741362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:22.741450 (+ 88us) client_negotiation.cc:769] Sending connection context
0806 12:26:22.741539 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 12:26:22.741719 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260806 12:26:22.742580 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.728216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.28.210.124:33415)
0806 12:26:22.728326 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.728330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.728343 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:26:22.728377 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.728380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.728436 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.728922 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.728975 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.729666 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.730617 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.730948 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.731324 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.733259 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.733263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.733266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.733290 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.734722 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.740289 (+ 5567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.740295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.740297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.740516 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.740985 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.740989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.740991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.741242 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.742171 (+ 929us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.742261 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.742402 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 12:26:22.743305 17863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } } }
I20260806 12:26:22.743577 17863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:26:22.745082 17714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:26:22.746305 17833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9: Bootstrap starting.
Aug 06 12:26:22 dist-test-slave-lgr5 krb5kdc[17482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.125@KRBTEST.COM
W20260806 12:26:22.751778 17880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260806 12:26:22.752238 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.740253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.28.210.125:36685)
0806 12:26:22.740414 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.740422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.740442 (+ 20us) server_negotiation.cc:407] Connection header received
0806 12:26:22.740512 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.740515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.740581 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.740698 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.742272 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.742984 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.744123 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.744435 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.744539 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.747733 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.747738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.747741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.747783 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.749764 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.750329 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.750334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.750336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.750435 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.750759 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.750762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.750764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.751018 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.751211 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.751916 (+ 705us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.752054 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 12:26:22.752365 17880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:26:22.754118 17833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:22.755080 17833 log.cc:824] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:22.757534 17833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9: No bootstrap required, opened a new log
I20260806 12:26:22.754495 17609 master_runner.cc:186] Error getting master registration for 127.28.210.124:33415: OK, instance_id { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" instance_seqno: 1786019182690748 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 12:26:22.757686 17609 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 12:26:22.759756 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:26:22.762094 17833 raft_consensus.cc:348] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } }
I20260806 12:26:22.762377 17833 raft_consensus.cc:374] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:22.762439 17833 raft_consensus.cc:729] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44df286eac6439eb4abf8c015dd81a9, State: Initialized, Role: FOLLOWER
I20260806 12:26:22.762948 17833 consensus_queue.cc:260] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } }
I20260806 12:26:22.768412 17833 sys_catalog.cc:564] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:26:22.773118 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.762239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.28.210.124:33415)
0806 12:26:22.762452 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.762458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.763252 (+ 794us) server_negotiation.cc:407] Connection header received
0806 12:26:22.763440 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.763456 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.763506 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.763557 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.764604 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.765382 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.766572 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.766924 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.766998 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.769006 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.769010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.769013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.769042 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.770728 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.771210 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.771214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.771216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.771309 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.771641 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.771644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.771646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.771866 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.772012 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.772825 (+ 813us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.772953 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260806 12:26:22.768356 17897 sys_catalog.cc:455] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } } }
I20260806 12:26:22.773386 17897 sys_catalog.cc:458] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:26:22.778430 17724 master_runner.cc:263] No leader master found from master e44df286eac6439eb4abf8c015dd81a9
I20260806 12:26:22.778507 17724 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 12:26:22.783414 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.762584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.28.210.126:35717)
0806 12:26:22.762701 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.762705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.768771 (+ 6066us) server_negotiation.cc:407] Connection header received
0806 12:26:22.773678 (+ 4907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.773682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.773739 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.773794 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.775147 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.775929 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.777076 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.777408 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.777517 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.779135 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.779139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.779142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.779171 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.781305 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.782035 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.782038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.782040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.782131 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.782415 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.782419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.782421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.782654 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.782753 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.783109 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.783239 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 12:26:22.784426 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:22.763213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.28.210.125:36685)
0806 12:26:22.763328 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:22.763332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:22.775355 (+ 12023us) server_negotiation.cc:407] Connection header received
0806 12:26:22.775641 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:22.775645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:22.775695 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:22.775745 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:22.776652 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.777352 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.778748 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:22.779074 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:22.779165 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:22.780769 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:22.780773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:22.780776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:22.780804 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:22.782325 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.782801 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.782805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.782807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.782894 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:22.783178 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:22.783181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:22.783183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:22.783388 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:22.783502 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:22.784138 (+ 636us) server_negotiation.cc:299] Negotiation successful
0806 12:26:22.784273 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 12:26:22.788002 17918 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:26:22.788127 17918 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:26:22.819473 17847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:26:22.819725 17847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } }
I20260806 12:26:22.821027 17847 leader_election.cc:288] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6296f00d82044e76b2999e98a5b4c1dd (127.28.210.125:36685), e44df286eac6439eb4abf8c015dd81a9 (127.28.210.124:33415)
I20260806 12:26:22.821842 17674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94e73341d90c4b8d922eac0f79d420ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6296f00d82044e76b2999e98a5b4c1dd" is_pre_election: true
I20260806 12:26:22.822234 17674 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94e73341d90c4b8d922eac0f79d420ca in term 0.
I20260806 12:26:22.822749 17523 leader_election.cc:302] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6296f00d82044e76b2999e98a5b4c1dd, 94e73341d90c4b8d922eac0f79d420ca; no voters:
I20260806 12:26:22.823657 17795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94e73341d90c4b8d922eac0f79d420ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44df286eac6439eb4abf8c015dd81a9" is_pre_election: true
I20260806 12:26:22.824031 17795 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94e73341d90c4b8d922eac0f79d420ca in term 0.
I20260806 12:26:22.824713 17847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:26:22.824812 17847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:26:22.824877 17847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:22.826284 17847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } }
I20260806 12:26:22.826678 17847 leader_election.cc:288] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [CANDIDATE]: Term 1 election: Requested vote from peers 6296f00d82044e76b2999e98a5b4c1dd (127.28.210.125:36685), e44df286eac6439eb4abf8c015dd81a9 (127.28.210.124:33415)
I20260806 12:26:22.827188 17674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94e73341d90c4b8d922eac0f79d420ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6296f00d82044e76b2999e98a5b4c1dd"
I20260806 12:26:22.827332 17674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:22.828962 17674 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94e73341d90c4b8d922eac0f79d420ca in term 1.
I20260806 12:26:22.829476 17523 leader_election.cc:302] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6296f00d82044e76b2999e98a5b4c1dd, 94e73341d90c4b8d922eac0f79d420ca; no voters:
I20260806 12:26:22.830101 17795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94e73341d90c4b8d922eac0f79d420ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44df286eac6439eb4abf8c015dd81a9"
I20260806 12:26:22.830276 17795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:22.831749 17795 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94e73341d90c4b8d922eac0f79d420ca in term 1.
I20260806 12:26:22.835271 17847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:26:22.835604 17847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [term 1 LEADER]: Becoming Leader. State: Replica: 94e73341d90c4b8d922eac0f79d420ca, State: Running, Role: LEADER
I20260806 12:26:22.836021 17847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } }
I20260806 12:26:22.837975 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94e73341d90c4b8d922eac0f79d420ca. Latest consensus state: current_term: 1 leader_uuid: "94e73341d90c4b8d922eac0f79d420ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } } }
I20260806 12:26:22.838092 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:22.838502 17922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:26:22.839491 17922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:26:22.850224 17795 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [term 1 FOLLOWER]: Refusing update from remote peer 94e73341d90c4b8d922eac0f79d420ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:26:22.850950 17920 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:26:22.855033 17897 sys_catalog.cc:455] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94e73341d90c4b8d922eac0f79d420ca. Latest consensus state: current_term: 1 leader_uuid: "94e73341d90c4b8d922eac0f79d420ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } } }
I20260806 12:26:22.855170 17897 sys_catalog.cc:458] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:26:22.858609 17674 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [term 1 FOLLOWER]: Refusing update from remote peer 94e73341d90c4b8d922eac0f79d420ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:26:22.859179 17920 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:26:22.863404 17863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94e73341d90c4b8d922eac0f79d420ca. Latest consensus state: current_term: 1 leader_uuid: "94e73341d90c4b8d922eac0f79d420ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } } }
I20260806 12:26:22.863533 17863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:26:22.872103 17920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94e73341d90c4b8d922eac0f79d420ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } } }
I20260806 12:26:22.872247 17920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:22.873485 17863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94e73341d90c4b8d922eac0f79d420ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } } }
I20260806 12:26:22.873611 17863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:26:22.873986 17925 mvcc.cc:205] Tried to move back new op lower bound from 7315534572946358272 to 7315534572899864576. Current Snapshot: MvccSnapshot[applied={T|T < 7315534572946358272}]
I20260806 12:26:22.876118 17929 mvcc.cc:205] Tried to move back new op lower bound from 7315534572946358272 to 7315534572899864576. Current Snapshot: MvccSnapshot[applied={T|T < 7315534572946358272}]
I20260806 12:26:22.876945 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94e73341d90c4b8d922eac0f79d420ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } } }
I20260806 12:26:22.877058 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:22.878116 17897 sys_catalog.cc:455] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94e73341d90c4b8d922eac0f79d420ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e73341d90c4b8d922eac0f79d420ca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35717 } } peers { permanent_uuid: "6296f00d82044e76b2999e98a5b4c1dd" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 36685 } } peers { permanent_uuid: "e44df286eac6439eb4abf8c015dd81a9" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 33415 } } }
I20260806 12:26:22.878237 17897 sys_catalog.cc:458] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:26:22.878930 17926 mvcc.cc:205] Tried to move back new op lower bound from 7315534572946358272 to 7315534572899864576. Current Snapshot: MvccSnapshot[applied={T|T < 7315534572946358272}]
I20260806 12:26:22.886085 17922 catalog_manager.cc:1347] Generated new cluster ID: f0d57b0f167c4d31953ecc00fe5b61dc
I20260806 12:26:22.886169 17922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:26:22.914045 17922 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:26:22.915458 17922 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:26:22.938349 17922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94e73341d90c4b8d922eac0f79d420ca: Generated new TSK 0
I20260806 12:26:22.939390 17922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:26:23.754182 17880 catalog_manager.cc:1259] Loaded cluster ID: f0d57b0f167c4d31953ecc00fe5b61dc
I20260806 12:26:23.754295 17880 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd: loading cluster ID for follower catalog manager: success
I20260806 12:26:23.756176 17880 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd: acquiring CA information for follower catalog manager: success
I20260806 12:26:23.757439 17880 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6296f00d82044e76b2999e98a5b4c1dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 12:26:23.789671 17918 catalog_manager.cc:1259] Loaded cluster ID: f0d57b0f167c4d31953ecc00fe5b61dc
I20260806 12:26:23.789758 17918 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9: loading cluster ID for follower catalog manager: success
I20260806 12:26:23.791471 17918 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9: acquiring CA information for follower catalog manager: success
I20260806 12:26:23.792659 17918 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e44df286eac6439eb4abf8c015dd81a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 12:26:24.009595 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 17499
I20260806 12:26:24.022766 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 17609
I20260806 12:26:24.037441 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 17724
2026-08-06T12:26:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3278 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:26:24 dist-test-slave-lgr5 krb5kdc[17946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:26:24 dist-test-slave-lgr5 krb5kdc[17946](info): set up 2 sockets
Aug 06 12:26:24 dist-test-slave-lgr5 krb5kdc[17946](info): commencing operation
krb5kdc: starting...
W20260806 12:26:25.181247 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:26:25 dist-test-slave-lgr5 krb5kdc[17946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:26:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:26:25.319731 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:43811
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:43811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:25.438732 17971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:25.439051 17971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:25.439097 17971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:25.443377 17971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:26:25.443472 17971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:25.443506 17971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:25.443544 17971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:26:25.443578 17971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:26:25.449366 17971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:43811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:43811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:25.450569 17971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:25.451737 17971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:25.461301 17976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:25.461404 17971 server_base.cc:1034] running on GCE node
W20260806 12:26:25.461301 17977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:25.461339 17980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:25.462216 17971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:25.463466 17971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:25.464648 17971 hybrid_clock.cc:648] HybridClock initialized: now 1786019185464621 us; error 38 us; skew 500 ppm
Aug 06 12:26:25 dist-test-slave-lgr5 krb5kdc[17946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019185, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:25.467613 17971 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:26:25.468725 17971 webserver.cc:460] Webserver started at http://127.28.210.126:38657/ using document root <none> and password file <none>
I20260806 12:26:25.469283 17971 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:25.469344 17971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:25.469525 17971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:26:25.471673 17971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "ddf931f8db4b4d74ba7fad9e34bce7b4"
format_stamp: "Formatted at 2026-08-06 12:26:25 on dist-test-slave-lgr5"
server_key: "f904de5a5744bf338a2dbe3130343e91"
server_key_iv: "e0ccbd470952229026e8ad3b7693afd6"
server_key_version: "encryptionkey@0"
I20260806 12:26:25.472303 17971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "ddf931f8db4b4d74ba7fad9e34bce7b4"
format_stamp: "Formatted at 2026-08-06 12:26:25 on dist-test-slave-lgr5"
server_key: "f904de5a5744bf338a2dbe3130343e91"
server_key_iv: "e0ccbd470952229026e8ad3b7693afd6"
server_key_version: "encryptionkey@0"
I20260806 12:26:25.476305 17971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:26:25.478922 17990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:25.480252 17971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 12:26:25.480389 17971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "ddf931f8db4b4d74ba7fad9e34bce7b4"
format_stamp: "Formatted at 2026-08-06 12:26:25 on dist-test-slave-lgr5"
server_key: "f904de5a5744bf338a2dbe3130343e91"
server_key_iv: "e0ccbd470952229026e8ad3b7693afd6"
server_key_version: "encryptionkey@0"
I20260806 12:26:25.480496 17971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:25.497104 17971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:25.500559 17971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:25.500728 17971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:25.507978 17971 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:43811
I20260806 12:26:25.508062 18073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:43811 every 8 connection(s)
I20260806 12:26:25.509116 17971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:26:25.512192 18074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:25.517689 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 17971
I20260806 12:26:25.517831 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:26:25.518123 29513 external_mini_cluster.cc:1428] Setting key d32ef4707d6e9519a007941b1a1e14bb
I20260806 12:26:25.518218 18074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4: Bootstrap starting.
I20260806 12:26:25.520889 18074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:25.521745 18074 log.cc:824] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:25.524008 18074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4: No bootstrap required, opened a new log
Aug 06 12:26:25 dist-test-slave-lgr5 krb5kdc[17946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:25.528484 18074 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf931f8db4b4d74ba7fad9e34bce7b4" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43811 } }
I20260806 12:26:25.528740 18074 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:25.528818 18074 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf931f8db4b4d74ba7fad9e34bce7b4, State: Initialized, Role: FOLLOWER
I20260806 12:26:25.529286 18074 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf931f8db4b4d74ba7fad9e34bce7b4" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43811 } }
I20260806 12:26:25.529426 18074 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:26:25.529495 18074 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:26:25.529590 18074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:25.530695 18074 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf931f8db4b4d74ba7fad9e34bce7b4" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43811 } }
I20260806 12:26:25.531080 18074 leader_election.cc:302] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf931f8db4b4d74ba7fad9e34bce7b4; no voters:
I20260806 12:26:25.531394 18074 leader_election.cc:288] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:26:25.531514 18084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:26:25.531764 18084 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [term 1 LEADER]: Becoming Leader. State: Replica: ddf931f8db4b4d74ba7fad9e34bce7b4, State: Running, Role: LEADER
I20260806 12:26:25.532222 18084 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf931f8db4b4d74ba7fad9e34bce7b4" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43811 } }
I20260806 12:26:25.532747 18074 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:26:25.533784 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:25.519491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.28.210.126:43811)
0806 12:26:25.519928 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:25.519958 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:25.520008 (+ 50us) server_negotiation.cc:407] Connection header received
0806 12:26:25.520721 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:25.520740 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:25.521125 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:25.521415 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:25.522323 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:25.523433 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:25.524618 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:25.525083 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:25.525215 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:25.528117 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:25.528125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:25.528145 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:25.528178 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:25.530027 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:25.530521 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:25.530526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:25.530531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:25.530652 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:25.530924 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:25.530926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:25.530928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:25.531398 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:25.531607 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:25.532457 (+ 850us) server_negotiation.cc:299] Negotiation successful
0806 12:26:25.532774 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":130,"threads_started":1}
I20260806 12:26:25.534004 18086 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddf931f8db4b4d74ba7fad9e34bce7b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf931f8db4b4d74ba7fad9e34bce7b4" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43811 } } }
I20260806 12:26:25.534152 18086 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:25.534523 18087 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf931f8db4b4d74ba7fad9e34bce7b4. Latest consensus state: current_term: 1 leader_uuid: "ddf931f8db4b4d74ba7fad9e34bce7b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf931f8db4b4d74ba7fad9e34bce7b4" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 43811 } } }
I20260806 12:26:25.534662 18087 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4 [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:25.537153 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 12:26:25.538264 18110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:26:25.538347 18110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:26:25.538440 18105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:26:25.539307 18105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:26:25.544797 18105 catalog_manager.cc:1347] Generated new cluster ID: ac99684d15324f30a41f7eb1b92cdf0a
I20260806 12:26:25.544879 18105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:26:25.572134 18105 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:26:25.573608 18105 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:26:25.592495 18105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddf931f8db4b4d74ba7fad9e34bce7b4: Generated new TSK 0
I20260806 12:26:25.593242 18105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:26:25.644506 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 17971
2026-08-06T12:26:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1612 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:26:25 dist-test-slave-lgr5 krb5kdc[18121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:26:25 dist-test-slave-lgr5 krb5kdc[18121](info): set up 2 sockets
Aug 06 12:26:25 dist-test-slave-lgr5 krb5kdc[18121](info): commencing operation
krb5kdc: starting...
W20260806 12:26:26.802121 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:26:26 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:26:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:26:26.936627 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:45431
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:45431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:27.056632 18139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:27.056959 18139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:27.057020 18139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:27.061223 18139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:26:27.061308 18139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:27.061335 18139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:27.061358 18139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:26:27.061388 18139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:26:27.067159 18139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:45431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:45431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:27.068380 18139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:27.069463 18139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:27.078747 18149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:27.078747 18144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:27.078807 18139 server_base.cc:1034] running on GCE node
W20260806 12:26:27.078749 18145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:27.079722 18139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:27.081135 18139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:27.082355 18139 hybrid_clock.cc:648] HybridClock initialized: now 1786019187082318 us; error 56 us; skew 500 ppm
Aug 06 12:26:27 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019187, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:27.089118 18139 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:26:27.090214 18139 webserver.cc:460] Webserver started at http://127.28.210.126:37141/ using document root <none> and password file <none>
I20260806 12:26:27.090787 18139 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:27.090857 18139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:27.091043 18139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:26:27.093261 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "0e231c1feced4d6491aea9110bbf86ad"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "1f2c440212b04359a87dfcaf6da1214f"
server_key_iv: "dab0aeef9ec6f74d698d4ed9e924cd7d"
server_key_version: "encryptionkey@0"
I20260806 12:26:27.093794 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "0e231c1feced4d6491aea9110bbf86ad"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "1f2c440212b04359a87dfcaf6da1214f"
server_key_iv: "dab0aeef9ec6f74d698d4ed9e924cd7d"
server_key_version: "encryptionkey@0"
I20260806 12:26:27.097786 18139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:26:27.100268 18158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:27.101979 18139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:27.102138 18139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "0e231c1feced4d6491aea9110bbf86ad"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "1f2c440212b04359a87dfcaf6da1214f"
server_key_iv: "dab0aeef9ec6f74d698d4ed9e924cd7d"
server_key_version: "encryptionkey@0"
I20260806 12:26:27.102322 18139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:27.134841 18139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:27.139086 18139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:27.139304 18139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:27.147969 18139 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:45431
I20260806 12:26:27.148350 18229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:45431 every 8 connection(s)
I20260806 12:26:27.149149 18139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:26:27.153246 18230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:27.156451 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 18139
I20260806 12:26:27.156589 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:26:27.156864 29513 external_mini_cluster.cc:1428] Setting key 35066e28389a69738257d685478b0b65
I20260806 12:26:27.161993 18230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Bootstrap starting.
I20260806 12:26:27.165096 18230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:27.166088 18230 log.cc:824] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Log is configured to *not* fsync() on all Append() calls
Aug 06 12:26:27 dist-test-slave-lgr5 krb5kdc[18121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:27.168804 18230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: No bootstrap required, opened a new log
I20260806 12:26:27.173660 18230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } }
I20260806 12:26:27.173918 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.158363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.28.210.126:45431)
0806 12:26:27.158812 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:27.158832 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:27.158869 (+ 37us) server_negotiation.cc:407] Connection header received
0806 12:26:27.159565 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:27.159585 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:27.159992 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:27.160371 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:27.161786 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.163249 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.164349 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.165093 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.165275 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:27.168237 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:27.168243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:27.168263 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:27.168298 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:27.170269 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:27.170732 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:27.170738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:27.170744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:27.170903 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:27.171159 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:27.171162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:27.171164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:27.171730 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:27.171941 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:27.172423 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 12:26:27.172756 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":131,"threads_started":1}
I20260806 12:26:27.173979 18230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:27.174036 18230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e231c1feced4d6491aea9110bbf86ad, State: Initialized, Role: FOLLOWER
I20260806 12:26:27.174587 18230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } }
I20260806 12:26:27.174759 18230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:26:27.174815 18230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:26:27.174932 18230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:27.176680 18230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } }
I20260806 12:26:27.177031 18230 leader_election.cc:302] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e231c1feced4d6491aea9110bbf86ad; no voters:
I20260806 12:26:27.177358 18230 leader_election.cc:288] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:26:27.177537 18235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:26:27.177772 18235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 1 LEADER]: Becoming Leader. State: Replica: 0e231c1feced4d6491aea9110bbf86ad, State: Running, Role: LEADER
I20260806 12:26:27.178112 18235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } }
I20260806 12:26:27.178846 18230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:26:27.179854 18238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e231c1feced4d6491aea9110bbf86ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } } }
I20260806 12:26:27.180038 18238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:27.180814 18240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e231c1feced4d6491aea9110bbf86ad. Latest consensus state: current_term: 1 leader_uuid: "0e231c1feced4d6491aea9110bbf86ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } } }
I20260806 12:26:27.181005 18240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:27.180876 18250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:26:27.184901 18250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:26:27.190917 18250 catalog_manager.cc:1347] Generated new cluster ID: c53e24f756ae4023bde097827e06fb3a
I20260806 12:26:27.191015 18250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:26:27.217787 18250 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:26:27.218816 18250 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:26:27.231303 18250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Generated new TSK 0
I20260806 12:26:27.232105 18250 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:26:27.271874 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:45431
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:27.390483 18271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:27.390793 18271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:27.390864 18271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:27.394980 18271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:27.395057 18271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:27.395140 18271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:26:27.400923 18271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:45431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:27.402216 18271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:27.403326 18271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:27.414669 18277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:27.414671 18283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:27.414669 18280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:27.414937 18271 server_base.cc:1034] running on GCE node
I20260806 12:26:27.415455 18271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:27.416191 18271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:27.417378 18271 hybrid_clock.cc:648] HybridClock initialized: now 1786019187417370 us; error 48 us; skew 500 ppm
Aug 06 12:26:27 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019187, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:27.420467 18271 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:26:27.421511 18271 webserver.cc:460] Webserver started at http://127.28.210.65:34321/ using document root <none> and password file <none>
I20260806 12:26:27.422055 18271 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:27.422113 18271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:27.422284 18271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:26:27.424407 18271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "0c361ecb1cfb48ebb7a4738fb07a304e"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "21f807b8fc3aee6c9b221f9a52478622"
server_key_iv: "731c1e35d04c2e379ca482beca85588e"
server_key_version: "encryptionkey@0"
I20260806 12:26:27.424948 18271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "0c361ecb1cfb48ebb7a4738fb07a304e"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "21f807b8fc3aee6c9b221f9a52478622"
server_key_iv: "731c1e35d04c2e379ca482beca85588e"
server_key_version: "encryptionkey@0"
I20260806 12:26:27.428750 18271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 12:26:27.431216 18293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:27.432595 18271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 12:26:27.432711 18271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "0c361ecb1cfb48ebb7a4738fb07a304e"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "21f807b8fc3aee6c9b221f9a52478622"
server_key_iv: "731c1e35d04c2e379ca482beca85588e"
server_key_version: "encryptionkey@0"
I20260806 12:26:27.432821 18271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:27.452966 18271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:27.455907 18271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:27.456112 18271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:27.456743 18271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:27.457590 18271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:27.457636 18271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:27.457679 18271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:27.457707 18271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:27.466740 18271 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:38601
I20260806 12:26:27.466802 18450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:38601 every 8 connection(s)
I20260806 12:26:27.467762 18271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:26:27.469451 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 18271
I20260806 12:26:27.469587 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:26:27.470070 29513 external_mini_cluster.cc:1428] Setting key 0bd22d92d610c446b10835b0786dac08
Aug 06 12:26:27 dist-test-slave-lgr5 krb5kdc[18121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019187, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:26:27.483989 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.469907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:58659 (local address 127.28.210.126:45431)
0806 12:26:27.470092 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:27.470097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:27.471131 (+ 1034us) server_negotiation.cc:407] Connection header received
0806 12:26:27.472733 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:27.472737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:27.472793 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:27.472883 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:27.474135 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.474908 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.476542 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.476779 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.476885 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:27.480012 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:27.480017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:27.480020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:27.480053 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:27.481735 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:27.482282 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:27.482286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:27.482288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:27.482375 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:27.482798 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:27.482802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:27.482804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:27.483010 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:27.483089 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:27.483734 (+ 645us) server_negotiation.cc:299] Negotiation successful
0806 12:26:27.483854 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 12:26:27.485085 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.470365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:58659)
0806 12:26:27.470946 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:27.471015 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:27.472505 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:27.473044 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:27.473051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:27.473432 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:27.473978 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:27.473995 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.475082 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.475085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:27.476370 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:27.476384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.476977 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.476981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:27.477186 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:27.477848 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:27.477880 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:27.479817 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:27.481877 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:27.481883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:27.481899 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:27.482175 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:27.482499 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:27.482502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:27.482503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:27.482696 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:27.483116 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:27.483122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:27.483431 (+ 309us) client_negotiation.cc:769] Sending connection context
0806 12:26:27.483677 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 12:26:27.483984 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:26:27.486493 18451 heartbeater.cc:347] Connected to a master server at 127.28.210.126:45431
I20260806 12:26:27.486742 18451 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:27.487278 18451 heartbeater.cc:511] Master 127.28.210.126:45431 requested a full tablet report, sending...
I20260806 12:26:27.488901 18180 ts_manager.cc:194] Registered new tserver with Master: 0c361ecb1cfb48ebb7a4738fb07a304e (127.28.210.65:38601)
I20260806 12:26:27.490077 18180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:58659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:26:27.504343 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:45431
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:27.623443 18459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:27.623740 18459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:27.623785 18459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:27.627885 18459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:27.627964 18459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:27.628023 18459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:26:27.633673 18459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:45431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:27.634773 18459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:27.635797 18459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:27.645790 18467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:27.645789 18464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:27.645955 18459 server_base.cc:1034] running on GCE node
W20260806 12:26:27.645810 18465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:27.646529 18459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:27.647156 18459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:27.648321 18459 hybrid_clock.cc:648] HybridClock initialized: now 1786019187648302 us; error 29 us; skew 500 ppm
Aug 06 12:26:27 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019187, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:27.651057 18459 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:26:27.651983 18459 webserver.cc:460] Webserver started at http://127.28.210.66:34893/ using document root <none> and password file <none>
I20260806 12:26:27.652607 18459 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:27.652678 18459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:27.652874 18459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:26:27.655068 18459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "fed7accba7534483aa9ddb1fadb2375b"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "c35618e9830cb0e4ddfd900ae7e325d3"
server_key_iv: "2706f7073b78662f39b9deb6a7788f10"
server_key_version: "encryptionkey@0"
I20260806 12:26:27.655651 18459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "fed7accba7534483aa9ddb1fadb2375b"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "c35618e9830cb0e4ddfd900ae7e325d3"
server_key_iv: "2706f7073b78662f39b9deb6a7788f10"
server_key_version: "encryptionkey@0"
I20260806 12:26:27.659706 18459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 12:26:27.662374 18475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:27.663815 18459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:26:27.663967 18459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "fed7accba7534483aa9ddb1fadb2375b"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "c35618e9830cb0e4ddfd900ae7e325d3"
server_key_iv: "2706f7073b78662f39b9deb6a7788f10"
server_key_version: "encryptionkey@0"
I20260806 12:26:27.664150 18459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:27.681823 18459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:27.685109 18459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:27.685297 18459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:27.685895 18459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:27.686745 18459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:27.686794 18459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:27.686847 18459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:27.686872 18459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:27.695604 18459 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:38743
I20260806 12:26:27.695636 18640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:38743 every 8 connection(s)
I20260806 12:26:27.696560 18459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:26:27.702791 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 18459
I20260806 12:26:27.702926 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:26:27.703159 29513 external_mini_cluster.cc:1428] Setting key e97c32c3a9269acef7d7ba20cdc90ff9
Aug 06 12:26:27 dist-test-slave-lgr5 krb5kdc[18121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019187, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:27.711380 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.698628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:60225 (local address 127.28.210.126:45431)
0806 12:26:27.698765 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:27.698769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:27.699623 (+ 854us) server_negotiation.cc:407] Connection header received
0806 12:26:27.701055 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:27.701059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:27.701118 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:27.701195 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:27.702398 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.702953 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.704198 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.704452 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.704555 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:27.707435 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:27.707440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:27.707443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:27.707482 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:27.709170 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:27.709750 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:27.709754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:27.709756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:27.709869 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:27.710209 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:27.710212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:27.710213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:27.710430 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:27.710524 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:27.711076 (+ 552us) server_negotiation.cc:299] Negotiation successful
0806 12:26:27.711209 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 12:26:27.712412 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.698999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:60225)
0806 12:26:27.699475 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:27.699520 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:27.700828 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:27.701383 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:27.701390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:27.701767 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:27.702252 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:27.702265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.703082 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.703085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:27.704047 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:27.704060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.704793 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.704796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:27.704972 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:27.705573 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:27.705591 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:27.707200 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:27.709312 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:27.709319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:27.709336 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:27.709638 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:27.709961 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:27.709963 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:27.709964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:27.710102 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:27.710542 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:27.710550 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:27.710825 (+ 275us) client_negotiation.cc:769] Sending connection context
0806 12:26:27.711018 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 12:26:27.711384 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":118,"threads_started":1}
I20260806 12:26:27.713810 18641 heartbeater.cc:347] Connected to a master server at 127.28.210.126:45431
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
I20260806 12:26:27.714092 18641 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:27.714674 18641 heartbeater.cc:511] Master 127.28.210.126:45431 requested a full tablet report, sending...
I20260806 12:26:27.715873 18180 ts_manager.cc:194] Registered new tserver with Master: fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743)
I20260806 12:26:27.716496 18180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:60225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:26:27.737773 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:45431
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:27.854451 18648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:27.854727 18648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:27.854786 18648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:27.858830 18648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:27.858907 18648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:27.858986 18648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:26:27.864313 18648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:45431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:27.865398 18648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:27.866441 18648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:27.876241 18656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:27.876241 18655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:27.876441 18648 server_base.cc:1034] running on GCE node
W20260806 12:26:27.876665 18658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:27.877120 18648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:27.877853 18648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:27.879024 18648 hybrid_clock.cc:648] HybridClock initialized: now 1786019187879003 us; error 34 us; skew 500 ppm
Aug 06 12:26:27 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019187, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:27.882081 18648 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:26:27.883122 18648 webserver.cc:460] Webserver started at http://127.28.210.67:37115/ using document root <none> and password file <none>
I20260806 12:26:27.883730 18648 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:27.883807 18648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:27.884028 18648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:26:27.886291 18648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "558843e5dfa681769ace9339cda1b0af"
server_key_iv: "f5a4706e3f6504cb7582531d89bbbd0f"
server_key_version: "encryptionkey@0"
I20260806 12:26:27.886916 18648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "558843e5dfa681769ace9339cda1b0af"
server_key_iv: "f5a4706e3f6504cb7582531d89bbbd0f"
server_key_version: "encryptionkey@0"
I20260806 12:26:27.891010 18648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 12:26:27.893568 18669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:27.894802 18648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 12:26:27.894936 18648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "558843e5dfa681769ace9339cda1b0af"
server_key_iv: "f5a4706e3f6504cb7582531d89bbbd0f"
server_key_version: "encryptionkey@0"
I20260806 12:26:27.895049 18648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:27.922640 18648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:27.926311 18648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:27.926527 18648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:27.927203 18648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:27.928237 18648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:27.928294 18648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:27.928354 18648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:27.928381 18648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:27.937677 18648 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:34435
I20260806 12:26:27.937696 18845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:34435 every 8 connection(s)
I20260806 12:26:27.938702 18648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:26:27.946571 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 18648
I20260806 12:26:27.946717 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:26:27.947036 29513 external_mini_cluster.cc:1428] Setting key 7fa269cff58cab5cb0e4b913e78b9a85
Aug 06 12:26:27 dist-test-slave-lgr5 krb5kdc[18121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019187, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:27.953609 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.940739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:36075 (local address 127.28.210.126:45431)
0806 12:26:27.940893 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:27.940898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:27.941860 (+ 962us) server_negotiation.cc:407] Connection header received
0806 12:26:27.943255 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:27.943258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:27.943310 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:27.943390 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:27.944732 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.945281 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.946523 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.946753 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.946854 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:27.949975 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:27.949978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:27.949981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:27.950010 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:27.951476 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:27.951988 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:27.951990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:27.951992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:27.952065 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:27.952423 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:27.952427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:27.952428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:27.952638 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:27.952737 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:27.953356 (+ 619us) server_negotiation.cc:299] Negotiation successful
0806 12:26:27.953475 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 12:26:27.954560 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.941170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:36075)
0806 12:26:27.941698 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:27.941746 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:27.943026 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:27.943596 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:27.943603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:27.943992 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:27.944581 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:27.944596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.945411 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.945414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:27.946386 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:27.946396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.946935 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.946940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:27.947161 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:27.947879 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:27.947900 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:27.949777 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:27.951603 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:27.951609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:27.951623 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:27.951908 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:27.952158 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:27.952161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:27.952163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:27.952305 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:27.952783 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:27.952789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:27.953084 (+ 295us) client_negotiation.cc:769] Sending connection context
0806 12:26:27.953257 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 12:26:27.953558 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":136,"threads_started":1}
I20260806 12:26:27.955889 18846 heartbeater.cc:347] Connected to a master server at 127.28.210.126:45431
I20260806 12:26:27.956212 18846 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:27.956799 18846 heartbeater.cc:511] Master 127.28.210.126:45431 requested a full tablet report, sending...
I20260806 12:26:27.958033 18180 ts_manager.cc:194] Registered new tserver with Master: d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435)
I20260806 12:26:27.958617 18180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:36075
I20260806 12:26:27.961511 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:26:27.971632 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.963676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.28.210.126:45431)
0806 12:26:27.963821 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:27.963826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:27.963977 (+ 151us) server_negotiation.cc:407] Connection header received
0806 12:26:27.964133 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:27.964135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:27.964183 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:27.964252 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:27.965014 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.965577 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.966466 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:27.966698 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:27.966792 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:27.968420 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:27.968424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:27.968426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:27.968461 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:27.969947 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:27.970421 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:27.970424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:27.970426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:27.970511 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:27.970703 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:27.970705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:27.970706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:27.970945 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:27.971047 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:27.971326 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 12:26:27.971460 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 12:26:27.982780 18180 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:26:27.986990 18180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:26:28.004294 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.997259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:34435 (local address 127.0.0.1:51230)
0806 12:26:27.998425 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:27.998494 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:27.998727 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:27.999502 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:27.999504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:27.999517 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:27.999757 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:27.999763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.001093 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.001097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.002243 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.002256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.003515 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.003519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.003659 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.003719 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 12:26:28.003870 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 12:26:28.004050 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"mutex_wait_us":36,"spinlock_wait_cycles":20864,"thread_start_us":124,"threads_started":1}
I20260806 12:26:28.004666 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.997927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.28.210.66:38743)
0806 12:26:27.998706 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:27.998725 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:27.998753 (+ 28us) server_negotiation.cc:407] Connection header received
0806 12:26:27.998989 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:27.998996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:27.999249 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:27.999428 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:27.999858 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.000956 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.002932 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.003622 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.003736 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.003893 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:28.004397 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 12:26:28.004524 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":158,"threads_started":1}
I20260806 12:26:28.005272 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.997097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:38743 (local address 127.0.0.1:44208)
0806 12:26:27.998425 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:27.998533 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:27.998728 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:27.999470 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:27.999477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:27.999507 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:27.999730 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:27.999739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.001143 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.001147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.002761 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.002774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.003964 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.003968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.004056 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.004110 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 12:26:28.004181 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 12:26:28.004246 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":224,"spinlock_wait_cycles":4096,"thread_start_us":123,"threads_started":1}
I20260806 12:26:28.005659 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.998023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.28.210.65:38601)
0806 12:26:27.999823 (+ 1800us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:27.999830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:27.999853 (+ 23us) server_negotiation.cc:407] Connection header received
0806 12:26:27.999929 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:27.999933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:28.000159 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:28.000315 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:28.000711 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.002318 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.003669 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.004596 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.004707 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.004866 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:28.005357 (+ 491us) server_negotiation.cc:299] Negotiation successful
0806 12:26:28.005504 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1630,"thread_start_us":91,"threads_started":1}
I20260806 12:26:28.005821 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.998170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51230 (local address 127.28.210.67:34435)
0806 12:26:27.999032 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:27.999043 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:27.999065 (+ 22us) server_negotiation.cc:407] Connection header received
0806 12:26:27.999145 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:27.999150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:27.999289 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:27.999428 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:27.999894 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.000957 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.002390 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.003358 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.003483 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.003689 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:28.003923 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 12:26:28.004057 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":165,"threads_started":1}
I20260806 12:26:28.005931 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:27.997491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:38601 (local address 127.0.0.1:38596)
0806 12:26:27.998437 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:27.998494 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:27.998727 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:28.000310 (+ 1583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:28.000313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:28.000336 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:28.000448 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.000454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.002422 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.002425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.003514 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.003526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.004932 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.004937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.005041 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.005070 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 12:26:28.005136 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 12:26:28.005212 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"spinlock_wait_cycles":206976,"thread_start_us":111,"threads_started":1}
I20260806 12:26:28.008975 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 5352865c77594550b6832a4f632e92b3 (DEFAULT_TABLE table=test-table [id=3c08ddae2d2440efbbe51f8923058ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:26:28.008981 18355 tablet_service.cc:1467] Processing CreateTablet for tablet 5352865c77594550b6832a4f632e92b3 (DEFAULT_TABLE table=test-table [id=3c08ddae2d2440efbbe51f8923058ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:26:28.008981 18548 tablet_service.cc:1467] Processing CreateTablet for tablet 5352865c77594550b6832a4f632e92b3 (DEFAULT_TABLE table=test-table [id=3c08ddae2d2440efbbe51f8923058ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:26:28.010257 18355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5352865c77594550b6832a4f632e92b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:28.010272 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5352865c77594550b6832a4f632e92b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:28.010607 18548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5352865c77594550b6832a4f632e92b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:28.017283 18869 tablet_bootstrap.cc:492] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Bootstrap starting.
I20260806 12:26:28.018864 18870 tablet_bootstrap.cc:492] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Bootstrap starting.
I20260806 12:26:28.018864 18871 tablet_bootstrap.cc:492] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b: Bootstrap starting.
I20260806 12:26:28.019807 18869 tablet_bootstrap.cc:654] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:28.021250 18869 log.cc:824] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:28.021327 18870 tablet_bootstrap.cc:654] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:28.021791 18871 tablet_bootstrap.cc:654] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:28.022260 18870 log.cc:824] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:28.022632 18871 log.cc:824] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:28.023640 18869 tablet_bootstrap.cc:492] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: No bootstrap required, opened a new log
I20260806 12:26:28.023881 18869 ts_tablet_manager.cc:1397] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 12:26:28.024355 18870 tablet_bootstrap.cc:492] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: No bootstrap required, opened a new log
I20260806 12:26:28.024539 18870 ts_tablet_manager.cc:1397] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 12:26:28.024632 18871 tablet_bootstrap.cc:492] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b: No bootstrap required, opened a new log
I20260806 12:26:28.024816 18871 ts_tablet_manager.cc:1397] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 12:26:28.027895 18870 raft_consensus.cc:348] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:28.027968 18869 raft_consensus.cc:348] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:28.028225 18870 raft_consensus.cc:374] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:28.028280 18870 raft_consensus.cc:729] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c361ecb1cfb48ebb7a4738fb07a304e, State: Initialized, Role: FOLLOWER
I20260806 12:26:28.028280 18869 raft_consensus.cc:374] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:28.028342 18869 raft_consensus.cc:729] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9cfe34ee75e4b5eb3b1eacb7b67c7db, State: Initialized, Role: FOLLOWER
I20260806 12:26:28.028554 18871 raft_consensus.cc:348] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:28.028793 18871 raft_consensus.cc:374] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:28.028841 18871 raft_consensus.cc:729] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed7accba7534483aa9ddb1fadb2375b, State: Initialized, Role: FOLLOWER
I20260806 12:26:28.028792 18870 consensus_queue.cc:260] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:28.028810 18869 consensus_queue.cc:260] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:28.029224 18871 consensus_queue.cc:260] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:28.029937 18870 ts_tablet_manager.cc:1428] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 12:26:28.029940 18869 ts_tablet_manager.cc:1428] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 12:26:28.029986 18846 heartbeater.cc:503] Master 127.28.210.126:45431 was elected leader, sending a full tablet report...
I20260806 12:26:28.030098 18871 ts_tablet_manager.cc:1428] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 12:26:28.030277 18451 heartbeater.cc:503] Master 127.28.210.126:45431 was elected leader, sending a full tablet report...
I20260806 12:26:28.030417 18641 heartbeater.cc:503] Master 127.28.210.126:45431 was elected leader, sending a full tablet report...
I20260806 12:26:28.141353 18877 raft_consensus.cc:482] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:26:28.141598 18877 raft_consensus.cc:504] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:28.142867 18877 leader_election.cc:288] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743), 0c361ecb1cfb48ebb7a4738fb07a304e (127.28.210.65:38601)
I20260806 12:26:28.146617 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:28.143191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:38601 (local address 127.28.210.67:34781)
0806 12:26:28.143303 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:28.143318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:28.143424 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:28.143697 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:28.143700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:28.143720 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:28.143837 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.143843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.144649 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.144652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.145489 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.145501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.146171 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.146175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.146288 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.146308 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 12:26:28.146375 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 12:26:28.146459 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":22}
I20260806 12:26:28.146617 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:28.143297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:34781 (local address 127.28.210.65:38601)
0806 12:26:28.143428 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:28.143433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:28.143449 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:26:28.143530 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:28.143533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:28.143576 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:28.143658 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:28.143951 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.144536 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.145623 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.146052 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.146171 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.146234 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:28.146406 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 12:26:28.146481 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 12:26:28.146868 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:28.143191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:38743 (local address 127.28.210.67:35509)
0806 12:26:28.143746 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:28.143758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:28.143853 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:28.144108 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:28.144111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:28.144127 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:28.144234 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.144239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.144962 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.144966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.145791 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.145800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.146453 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.146456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.146556 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.146577 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 12:26:28.146641 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 12:26:28.146711 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":144,"thread_start_us":195,"threads_started":1}
I20260806 12:26:28.146854 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:28.143298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:35509 (local address 127.28.210.66:38743)
0806 12:26:28.143426 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:28.143430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:28.143847 (+ 417us) server_negotiation.cc:407] Connection header received
0806 12:26:28.143945 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:28.143948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:28.143995 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:28.144062 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:28.144339 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.144855 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.145923 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.146338 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.146449 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.146517 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:28.146667 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 12:26:28.146741 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 12:26:28.147616 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5352865c77594550b6832a4f632e92b3" candidate_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" is_pre_election: true
I20260806 12:26:28.147681 18576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5352865c77594550b6832a4f632e92b3" candidate_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed7accba7534483aa9ddb1fadb2375b" is_pre_election: true
I20260806 12:26:28.148023 18387 raft_consensus.cc:2454] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9cfe34ee75e4b5eb3b1eacb7b67c7db in term 0.
I20260806 12:26:28.148021 18576 raft_consensus.cc:2454] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9cfe34ee75e4b5eb3b1eacb7b67c7db in term 0.
I20260806 12:26:28.148663 18670 leader_election.cc:302] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c361ecb1cfb48ebb7a4738fb07a304e, d9cfe34ee75e4b5eb3b1eacb7b67c7db; no voters:
I20260806 12:26:28.149026 18877 raft_consensus.cc:2781] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:26:28.149115 18877 raft_consensus.cc:482] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:26:28.149166 18877 raft_consensus.cc:3037] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:28.150384 18877 raft_consensus.cc:504] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:28.151261 18576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5352865c77594550b6832a4f632e92b3" candidate_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed7accba7534483aa9ddb1fadb2375b"
I20260806 12:26:28.151261 18387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5352865c77594550b6832a4f632e92b3" candidate_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e"
I20260806 12:26:28.151428 18576 raft_consensus.cc:3037] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:28.151428 18387 raft_consensus.cc:3037] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:28.151614 18877 leader_election.cc:288] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [CANDIDATE]: Term 1 election: Requested vote from peers fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743), 0c361ecb1cfb48ebb7a4738fb07a304e (127.28.210.65:38601)
I20260806 12:26:28.152948 18576 raft_consensus.cc:2454] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9cfe34ee75e4b5eb3b1eacb7b67c7db in term 1.
I20260806 12:26:28.152948 18387 raft_consensus.cc:2454] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9cfe34ee75e4b5eb3b1eacb7b67c7db in term 1.
I20260806 12:26:28.153399 18672 leader_election.cc:302] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9cfe34ee75e4b5eb3b1eacb7b67c7db, fed7accba7534483aa9ddb1fadb2375b; no voters:
I20260806 12:26:28.153690 18877 raft_consensus.cc:2781] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:26:28.153959 18877 raft_consensus.cc:686] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 1 LEADER]: Becoming Leader. State: Replica: d9cfe34ee75e4b5eb3b1eacb7b67c7db, State: Running, Role: LEADER
I20260806 12:26:28.154353 18877 consensus_queue.cc:237] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:28.158322 18177 catalog_manager.cc:5495] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db reported cstate change: term changed from 0 to 1, leader changed from <none> to d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67). New cstate: current_term: 1 leader_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } health_report { overall_health: UNKNOWN } } }
W20260806 12:26:28.189777 18847 tablet.cc:2369] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:26:28.197623 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:28.193972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.28.210.67:34435)
0806 12:26:28.194093 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:28.194097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:28.194111 (+ 14us) server_negotiation.cc:407] Connection header received
0806 12:26:28.194191 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:28.194193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:28.194249 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:28.194341 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:26:28.194606 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.195137 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.195976 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.196239 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.196362 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.196673 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:26:28.197155 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:26:28.197263 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:28.197368 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 12:26:28.197455 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 12:26:28.197767 18644 tablet.cc:2369] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:26:28.204038 18576 raft_consensus.cc:1261] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b [term 1 FOLLOWER]: Refusing update from remote peer d9cfe34ee75e4b5eb3b1eacb7b67c7db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:26:28.204038 18387 raft_consensus.cc:1261] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Refusing update from remote peer d9cfe34ee75e4b5eb3b1eacb7b67c7db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:26:28.204910 18877 consensus_queue.cc:1035] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:26:28.205137 18885 consensus_queue.cc:1035] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:26:28.214076 18890 mvcc.cc:205] Tried to move back new op lower bound from 7315534594875011072 to 7315534594683662336. Current Snapshot: MvccSnapshot[applied={T|T < 7315534594875011072}]
I20260806 12:26:28.215924 18891 mvcc.cc:205] Tried to move back new op lower bound from 7315534594875011072 to 7315534594683662336. Current Snapshot: MvccSnapshot[applied={T|T < 7315534594875011072 or (T in {7315534594875011072})}]
I20260806 12:26:28.222693 18177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:26:28.222872 18177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:26:28.225417 18177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Sending DeleteTablet for 3 replicas of tablet 5352865c77594550b6832a4f632e92b3
I20260806 12:26:28.226307 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 5352865c77594550b6832a4f632e92b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:26:28 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:51230
I20260806 12:26:28.226310 18548 tablet_service.cc:1514] Processing DeleteTablet for tablet 5352865c77594550b6832a4f632e92b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:26:28 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:44208
I20260806 12:26:28.226598 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 18271
I20260806 12:26:28.236438 18904 tablet_replica.cc:330] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b: stopping tablet replica
I20260806 12:26:28.237007 18904 raft_consensus.cc:2229] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:26:28.237506 18904 raft_consensus.cc:2258] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:26:28.241468 18904 ts_tablet_manager.cc:1905] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:26:28.243103 18905 tablet_replica.cc:330] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: stopping tablet replica
I20260806 12:26:28.243495 18905 raft_consensus.cc:2229] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:26:28.243916 18905 raft_consensus.cc:2258] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:26:28.244539 18904 ts_tablet_manager.cc:1918] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:26:28.244648 18904 log.cc:1196] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/5352865c77594550b6832a4f632e92b3
I20260806 12:26:28.244978 18904 ts_tablet_manager.cc:1939] T 5352865c77594550b6832a4f632e92b3 P fed7accba7534483aa9ddb1fadb2375b: Deleting consensus metadata
I20260806 12:26:28.245716 18905 ts_tablet_manager.cc:1905] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:26:28.246381 18161 catalog_manager.cc:4841] TS fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743): tablet 5352865c77594550b6832a4f632e92b3 (table test-table [id=3c08ddae2d2440efbbe51f8923058ea2]) successfully deleted
I20260806 12:26:28.248818 18905 ts_tablet_manager.cc:1918] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:26:28.248939 18905 log.cc:1196] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/5352865c77594550b6832a4f632e92b3
I20260806 12:26:28.249337 18905 ts_tablet_manager.cc:1939] T 5352865c77594550b6832a4f632e92b3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Deleting consensus metadata
I20260806 12:26:28.250624 18162 catalog_manager.cc:4841] TS d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435): tablet 5352865c77594550b6832a4f632e92b3 (table test-table [id=3c08ddae2d2440efbbe51f8923058ea2]) successfully deleted
W20260806 12:26:28.251649 18159 connection.cc:535] client connection to 127.28.210.65:38601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 12:26:28.251791 18159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 12:26:28.252545 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 18459
I20260806 12:26:28.252926 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:28.252583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:38601 (local address 127.0.0.1:38600)
0806 12:26:28.252776 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:28.252838 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:38601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260806 12:26:28.270578 18159 catalog_manager.cc:4584] TS 0c361ecb1cfb48ebb7a4738fb07a304e (127.28.210.65:38601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5352865c77594550b6832a4f632e92b3: Network error: Client connection negotiation failed: client connection to 127.28.210.65:38601: connect: Connection refused (error 111)
I20260806 12:26:28.278327 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 18648
I20260806 12:26:28.293437 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 18139
I20260806 12:26:28.306146 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:45431
--webserver_interface=127.28.210.126
--webserver_port=37141
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:45431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:28.422137 18907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:28.422463 18907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:28.422506 18907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:28.426735 18907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:26:28.426832 18907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:28.426852 18907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:28.426868 18907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:26:28.426884 18907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:26:28.432840 18907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:45431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:45431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:28.434048 18907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:28.435141 18907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:28.450227 18920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:28.450256 18921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:28.450266 18924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:28.450440 18907 server_base.cc:1034] running on GCE node
I20260806 12:26:28.451243 18907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:28.452519 18907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:28.453706 18907 hybrid_clock.cc:648] HybridClock initialized: now 1786019188453687 us; error 37 us; skew 500 ppm
Aug 06 12:26:28 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:28.456897 18907 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:26:28.457993 18907 webserver.cc:460] Webserver started at http://127.28.210.126:37141/ using document root <none> and password file <none>
I20260806 12:26:28.458630 18907 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:28.458729 18907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:28.463028 18907 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:26:28.465499 18932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:28.466979 18907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 12:26:28.467119 18907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "0e231c1feced4d6491aea9110bbf86ad"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "1f2c440212b04359a87dfcaf6da1214f"
server_key_iv: "dab0aeef9ec6f74d698d4ed9e924cd7d"
server_key_version: "encryptionkey@0"
I20260806 12:26:28.467765 18907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:28.478078 18907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:28.481601 18907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:28.481806 18907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:28.489334 18907 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:45431
I20260806 12:26:28.489651 19014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:45431 every 8 connection(s)
I20260806 12:26:28.490604 18907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:26:28.494029 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 18907
I20260806 12:26:28.494626 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:38601
--local_ip_for_outbound_sockets=127.28.210.65
--tserver_master_addrs=127.28.210.126:45431
--webserver_port=34321
--webserver_interface=127.28.210.65
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:26:28.496894 19015 sys_catalog.cc:263] Verifying existing consensus state
I20260806 12:26:28.499037 19015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Bootstrap starting.
I20260806 12:26:28.519224 19015 log.cc:824] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:28.529870 19015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 12:26:28.530329 19015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Bootstrap complete.
I20260806 12:26:28.534584 19015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } }
I20260806 12:26:28.535050 19015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e231c1feced4d6491aea9110bbf86ad, State: Initialized, Role: FOLLOWER
I20260806 12:26:28.535583 19015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } }
I20260806 12:26:28.535707 19015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:26:28.535781 19015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:26:28.535880 19015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 1 FOLLOWER]: Advancing to term 2
I20260806 12:26:28.538007 19015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } }
I20260806 12:26:28.538360 19015 leader_election.cc:302] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e231c1feced4d6491aea9110bbf86ad; no voters:
I20260806 12:26:28.538674 19015 leader_election.cc:288] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 12:26:28.538787 19024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 2 FOLLOWER]: Leader election won for term 2
I20260806 12:26:28.539019 19024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 2 LEADER]: Becoming Leader. State: Replica: 0e231c1feced4d6491aea9110bbf86ad, State: Running, Role: LEADER
I20260806 12:26:28.539350 19024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } }
I20260806 12:26:28.540035 19015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:26:28.540536 19024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e231c1feced4d6491aea9110bbf86ad. Latest consensus state: current_term: 2 leader_uuid: "0e231c1feced4d6491aea9110bbf86ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } } }
I20260806 12:26:28.540654 19024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:28.540961 19026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0e231c1feced4d6491aea9110bbf86ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } } }
I20260806 12:26:28.541124 19026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:28.541136 19031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:26:28.546144 19031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:26:28.546797 19031 catalog_manager.cc:1259] Loaded cluster ID: c53e24f756ae4023bde097827e06fb3a
I20260806 12:26:28.546867 19031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:26:28.548014 19031 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 12:26:28.550787 19031 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 12:26:28.556032 19031 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:26:28.556838 19031 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Loaded TSK: 0
I20260806 12:26:28.557801 19031 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 12:26:28.627661 19017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:28.627954 19017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:28.628019 19017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:28.632117 19017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:28.632187 19017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:28.632269 19017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:26:28.637739 19017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:38601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=34321
--tserver_master_addrs=127.28.210.126:45431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:28.638782 19017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:28.639786 19017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:28.650665 19065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:28.650727 19069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:28.650702 19066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:28.651081 19017 server_base.cc:1034] running on GCE node
I20260806 12:26:28.651638 19017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:28.652449 19017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:28.653657 19017 hybrid_clock.cc:648] HybridClock initialized: now 1786019188653621 us; error 47 us; skew 500 ppm
Aug 06 12:26:28 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:28.657188 19017 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:26:28.658358 19017 webserver.cc:460] Webserver started at http://127.28.210.65:34321/ using document root <none> and password file <none>
I20260806 12:26:28.659113 19017 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:28.659235 19017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:28.664717 19017 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:26:28.667619 19080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:28.669278 19017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:28.669430 19017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "0c361ecb1cfb48ebb7a4738fb07a304e"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "21f807b8fc3aee6c9b221f9a52478622"
server_key_iv: "731c1e35d04c2e379ca482beca85588e"
server_key_version: "encryptionkey@0"
I20260806 12:26:28.670209 19017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:28.686400 19017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:28.690105 19017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:28.690311 19017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:28.691039 19017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:28.692378 19092 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 12:26:28.696764 19017 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 12:26:28.696842 19017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 12:26:28.696914 19017 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 12:26:28.699054 19017 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 12:26:28.699115 19017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 12:26:28.699436 19092 tablet_bootstrap.cc:492] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Bootstrap starting.
I20260806 12:26:28.709720 19017 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:38601
I20260806 12:26:28.710140 19248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:38601 every 8 connection(s)
I20260806 12:26:28.710942 19017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:26:28.714018 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 19017
I20260806 12:26:28.714401 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:38743
--local_ip_for_outbound_sockets=127.28.210.66
--tserver_master_addrs=127.28.210.126:45431
--webserver_port=34893
--webserver_interface=127.28.210.66
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:26:28.721009 19092 log.cc:824] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Log is configured to *not* fsync() on all Append() calls
Aug 06 12:26:28 dist-test-slave-lgr5 krb5kdc[18121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:28.730301 19092 tablet_bootstrap.cc:492] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 12:26:28.730841 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:28.712940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:34343)
0806 12:26:28.713610 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:28.713671 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:28.714841 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:28.716239 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:28.716249 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:28.716778 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:28.717540 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.717563 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.718980 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.718984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.720277 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.720291 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.721406 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.721410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.721647 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.722407 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:28.722429 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:28.724478 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:28.727193 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:28.727199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:28.727214 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:28.727607 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:28.728071 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:28.728074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:28.728098 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:28.728277 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:28.729007 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:28.729013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:28.729311 (+ 298us) client_negotiation.cc:769] Sending connection context
0806 12:26:28.729506 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 12:26:28.729849 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":127,"threads_started":1}
I20260806 12:26:28.730904 19092 tablet_bootstrap.cc:492] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Bootstrap complete.
I20260806 12:26:28.731353 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:28.712987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:34343 (local address 127.28.210.126:45431)
0806 12:26:28.713387 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:28.713437 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:28.713808 (+ 371us) server_negotiation.cc:407] Connection header received
0806 12:26:28.715325 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:28.715340 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:28.715747 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:28.716154 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:28.717766 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.718797 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.720447 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.721229 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.721454 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.724774 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:28.724783 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:28.724801 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:28.724843 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:28.727033 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:28.727765 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:28.727773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:28.727779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:28.727961 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:28.728394 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:28.728398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:28.728400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:28.728886 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:28.729039 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:28.729679 (+ 640us) server_negotiation.cc:299] Negotiation successful
0806 12:26:28.730012 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":125,"threads_started":1}
I20260806 12:26:28.732239 19092 ts_tablet_manager.cc:1397] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.007s
I20260806 12:26:28.734156 19249 heartbeater.cc:347] Connected to a master server at 127.28.210.126:45431
I20260806 12:26:28.734427 19249 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:28.735059 19249 heartbeater.cc:511] Master 127.28.210.126:45431 requested a full tablet report, sending...
I20260806 12:26:28.735052 19092 raft_consensus.cc:348] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:28.735523 19092 raft_consensus.cc:729] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c361ecb1cfb48ebb7a4738fb07a304e, State: Initialized, Role: FOLLOWER
I20260806 12:26:28.735971 19092 consensus_queue.cc:260] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:28.737028 19092 ts_tablet_manager.cc:1428] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 12:26:28.738602 18958 ts_manager.cc:194] Registered new tserver with Master: 0c361ecb1cfb48ebb7a4738fb07a304e (127.28.210.65:38601)
I20260806 12:26:28.742069 18958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:34343
I20260806 12:26:28.745224 19249 heartbeater.cc:503] Master 127.28.210.126:45431 was elected leader, sending a full tablet report...
Aug 06 12:26:28 dist-test-slave-lgr5 krb5kdc[18121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.65@KRBTEST.COM
I20260806 12:26:28.753575 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:28.742504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.28.210.65:38601)
0806 12:26:28.742806 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:28.742818 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:28.742845 (+ 27us) server_negotiation.cc:407] Connection header received
0806 12:26:28.742924 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:28.742929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:28.743143 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:28.743294 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:28.744152 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.745185 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.746507 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.747023 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.747138 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.749796 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:28.749807 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:28.749819 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:28.749844 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:28.751440 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:28.752024 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:28.752029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:28.752031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:28.752172 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:28.752457 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:28.752459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:28.752460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:28.752754 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:28.752858 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:28.753260 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 12:26:28.753412 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":107,"threads_started":1}
I20260806 12:26:28.753626 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:28.741672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:38601 (local address 127.0.0.1:38608)
0806 12:26:28.742114 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:28.742163 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:28.742300 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:28.743324 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:28.743330 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:28.743795 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:28.743947 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.743957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.745380 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.745384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.746339 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.746353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.747162 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.747166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.747290 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.747884 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:28.747900 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:28.749619 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:28.751590 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:28.751602 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:28.751608 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:28.751918 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:28.752256 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:28.752260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:28.752262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:28.752366 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:28.752871 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:28.752878 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:28.753031 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 12:26:28.753200 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 12:26:28.753447 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":138,"threads_started":1}
I20260806 12:26:28.754765 19156 tablet_service.cc:1514] Processing DeleteTablet for tablet 5352865c77594550b6832a4f632e92b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:26:28 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:38608
I20260806 12:26:28.755846 19263 tablet_replica.cc:330] stopping tablet replica
I20260806 12:26:28.756210 19263 raft_consensus.cc:2229] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:26:28.756485 19263 raft_consensus.cc:2258] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:26:28.757483 19263 ts_tablet_manager.cc:1905] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:26:28.761089 19263 ts_tablet_manager.cc:1918] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:26:28.761205 19263 log.cc:1196] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal/wals/5352865c77594550b6832a4f632e92b3
I20260806 12:26:28.761452 19263 ts_tablet_manager.cc:1939] T 5352865c77594550b6832a4f632e92b3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Deleting consensus metadata
I20260806 12:26:28.763520 18933 catalog_manager.cc:4841] TS 0c361ecb1cfb48ebb7a4738fb07a304e (127.28.210.65:38601): tablet 5352865c77594550b6832a4f632e92b3 (table test-table [id=3c08ddae2d2440efbbe51f8923058ea2]) successfully deleted
W20260806 12:26:28.840072 19255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:28.840409 19255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:28.840456 19255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:28.844681 19255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:28.844754 19255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:28.844816 19255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:26:28.850428 19255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:38743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=34893
--tserver_master_addrs=127.28.210.126:45431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:28.851657 19255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:28.852837 19255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:26:28.863770 19255 server_base.cc:1034] running on GCE node
W20260806 12:26:28.863723 19271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:28.863723 19273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:28.863723 19270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:28.864428 19255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:28.865202 19255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:28.866400 19255 hybrid_clock.cc:648] HybridClock initialized: now 1786019188866379 us; error 33 us; skew 500 ppm
Aug 06 12:26:28 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:28.869207 19255 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:26:28.870107 19255 webserver.cc:460] Webserver started at http://127.28.210.66:34893/ using document root <none> and password file <none>
I20260806 12:26:28.870692 19255 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:28.870774 19255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:28.874953 19255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:26:28.877274 19288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:28.878628 19255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:28.878757 19255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "fed7accba7534483aa9ddb1fadb2375b"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "c35618e9830cb0e4ddfd900ae7e325d3"
server_key_iv: "2706f7073b78662f39b9deb6a7788f10"
server_key_version: "encryptionkey@0"
I20260806 12:26:28.879341 19255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:28.898504 19255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:28.902064 19255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:28.902241 19255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:28.902856 19255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:28.903754 19255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:28.903801 19255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:28.903846 19255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:28.903875 19255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:28.912827 19255 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:38743
I20260806 12:26:28.912854 19466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:38743 every 8 connection(s)
I20260806 12:26:28.913944 19255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
Aug 06 12:26:28 dist-test-slave-lgr5 krb5kdc[18121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019188, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:28.924228 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 19255
I20260806 12:26:28.924664 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:34435
--local_ip_for_outbound_sockets=127.28.210.67
--tserver_master_addrs=127.28.210.126:45431
--webserver_port=37115
--webserver_interface=127.28.210.67
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:26:28.928058 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:28.915901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:36381 (local address 127.28.210.126:45431)
0806 12:26:28.916070 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:28.916075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:28.917020 (+ 945us) server_negotiation.cc:407] Connection header received
0806 12:26:28.918492 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:28.918495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:28.918569 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:28.918645 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:28.919959 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.920564 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.921748 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.921990 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.922080 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.924650 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:28.924654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:28.924656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:28.924683 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:28.926029 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:28.926521 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:28.926525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:28.926527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:28.926605 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:28.926927 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:28.926930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:28.926932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:28.927123 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:28.927211 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:28.927736 (+ 525us) server_negotiation.cc:299] Negotiation successful
0806 12:26:28.927884 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 12:26:28.928970 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:28.916350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:36381)
0806 12:26:28.916855 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:28.916900 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:28.918221 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:28.918862 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:28.918870 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:28.919278 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:28.919795 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.919810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.920702 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.920705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.921617 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:28.921628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:28.922144 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:28.922148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:28.922314 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:28.922891 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:28.922915 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:28.924465 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:28.926147 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:28.926154 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:28.926171 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:28.926424 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:28.926690 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:28.926693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:28.926695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:28.926836 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:28.927229 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:28.927237 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:28.927497 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 12:26:28.927676 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 12:26:28.928039 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260806 12:26:28.930569 19467 heartbeater.cc:347] Connected to a master server at 127.28.210.126:45431
I20260806 12:26:28.930830 19467 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:28.931370 19467 heartbeater.cc:511] Master 127.28.210.126:45431 requested a full tablet report, sending...
I20260806 12:26:28.932581 18958 ts_manager.cc:194] Registered new tserver with Master: fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743)
I20260806 12:26:28.933373 18958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:36381
W20260806 12:26:29.041882 19471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:29.042166 19471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:29.042214 19471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:29.046267 19471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:29.046340 19471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:29.046401 19471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:26:29.051869 19471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:34435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=37115
--tserver_master_addrs=127.28.210.126:45431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:29.052966 19471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:29.054030 19471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:29.064262 19480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:29.064265 19481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:29.064265 19483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:29.064425 19471 server_base.cc:1034] running on GCE node
I20260806 12:26:29.065073 19471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:29.065816 19471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:29.067003 19471 hybrid_clock.cc:648] HybridClock initialized: now 1786019189066970 us; error 48 us; skew 500 ppm
Aug 06 12:26:29 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019189, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:29.070047 19471 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:26:29.071154 19471 webserver.cc:460] Webserver started at http://127.28.210.67:37115/ using document root <none> and password file <none>
I20260806 12:26:29.071760 19471 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:29.071846 19471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:29.076545 19471 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:29.079080 19495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:29.080668 19471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:29.080850 19471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "558843e5dfa681769ace9339cda1b0af"
server_key_iv: "f5a4706e3f6504cb7582531d89bbbd0f"
server_key_version: "encryptionkey@0"
I20260806 12:26:29.081540 19471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:29.097573 19471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:29.100800 19471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:29.101023 19471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:29.101671 19471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:29.102561 19471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:29.102610 19471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:29.102664 19471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:29.102694 19471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:29.111532 19471 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:34435
I20260806 12:26:29.111562 19652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:34435 every 8 connection(s)
I20260806 12:26:29.112622 19471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:26:29.122385 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 19471
Aug 06 12:26:29 dist-test-slave-lgr5 krb5kdc[18121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019189, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:29.127421 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.114496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:50435 (local address 127.28.210.126:45431)
0806 12:26:29.114680 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:29.114685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:29.115609 (+ 924us) server_negotiation.cc:407] Connection header received
0806 12:26:29.117090 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:29.117093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:29.117149 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:29.117230 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:29.118558 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.119111 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.120362 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.120598 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.120704 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.123528 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:29.123532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:29.123536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:29.123568 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:29.125091 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:29.125715 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:29.125720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:29.125722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:29.125822 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:29.126165 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:29.126168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:29.126170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:29.126390 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:29.126467 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:29.127105 (+ 638us) server_negotiation.cc:299] Negotiation successful
0806 12:26:29.127243 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 12:26:29.128456 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.114918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:50435)
0806 12:26:29.115444 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:29.115489 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:29.116840 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:29.117485 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:29.117495 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:29.117875 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:29.118387 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:29.118403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.119238 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.119241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:29.120228 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:29.120239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.120778 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.120781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:29.120954 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.121558 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:29.121576 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:29.123291 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:29.125220 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:29.125227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:29.125246 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:29.125582 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:29.125925 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:29.125927 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:29.125929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:29.126084 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:29.126485 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:29.126492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:29.126840 (+ 348us) client_negotiation.cc:769] Sending connection context
0806 12:26:29.127051 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 12:26:29.127391 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":115,"threads_started":1}
I20260806 12:26:29.129657 19653 heartbeater.cc:347] Connected to a master server at 127.28.210.126:45431
I20260806 12:26:29.129905 19653 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:29.130525 19653 heartbeater.cc:511] Master 127.28.210.126:45431 requested a full tablet report, sending...
I20260806 12:26:29.131217 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.123198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.28.210.126:45431)
0806 12:26:29.123554 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:29.123557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:29.123607 (+ 50us) server_negotiation.cc:407] Connection header received
0806 12:26:29.123741 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:29.123743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:29.123781 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:29.123844 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:29.124730 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.125300 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.126292 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.126595 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.126692 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.128120 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:29.128125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:29.128128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:29.128153 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:29.129607 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:29.130030 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:29.130034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:29.130035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:29.130127 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:29.130359 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:29.130362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:29.130364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:29.130585 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:29.130785 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:29.130936 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 12:26:29.131062 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":255,"thread_start_us":123,"threads_started":1}
I20260806 12:26:29.131980 18958 ts_manager.cc:194] Registered new tserver with Master: d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435)
I20260806 12:26:29.132584 18958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:50435
I20260806 12:26:29.133285 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:26:29.137933 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.133986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49568 (local address 127.28.210.126:45431)
0806 12:26:29.134107 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:29.134111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:29.134382 (+ 271us) server_negotiation.cc:407] Connection header received
0806 12:26:29.134526 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:29.134528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:29.134625 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:29.134709 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:26:29.134936 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.135450 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.136440 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.136674 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.136755 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.137040 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:26:29.137474 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:26:29.137569 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:29.137708 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 12:26:29.137773 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 12:26:29.138797 18956 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:26:29.142343 18956 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:26:29.150823 19156 tablet_service.cc:1467] Processing CreateTablet for tablet eaed3fb47d954d7da720fffb711251a3 (DEFAULT_TABLE table=test-table [id=7f6a38081a084123b1c51cd830d0baf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:26:29.151573 19156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaed3fb47d954d7da720fffb711251a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:29.154400 19092 tablet_bootstrap.cc:492] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Bootstrap starting.
I20260806 12:26:29.155400 19092 tablet_bootstrap.cc:654] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:29.155493 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.149700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:34435 (local address 127.0.0.1:51256)
0806 12:26:29.149806 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:29.149822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:29.149908 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:29.151038 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:29.151041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:29.151060 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:29.151173 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:29.151180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.152697 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.152700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:29.153958 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:29.153970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.155013 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.155027 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:29.155130 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.155176 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 12:26:29.155249 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 12:26:29.155337 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":40}
I20260806 12:26:29.155660 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.150007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.28.210.67:34435)
0806 12:26:29.150582 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:29.150594 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:29.150619 (+ 25us) server_negotiation.cc:407] Connection header received
0806 12:26:29.150694 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:29.150699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:29.150879 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:29.151031 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:29.151305 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.152571 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.154114 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.154890 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.155003 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.155177 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:29.155420 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 12:26:29.155526 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":207,"threads_started":1}
I20260806 12:26:29.156648 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.150061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:38743 (local address 127.0.0.1:44218)
0806 12:26:29.150367 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:29.150380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:29.150469 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:29.151213 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:29.151216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:29.151234 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:29.151335 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:29.151341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.153021 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.153025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:29.154162 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:29.154170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.156252 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.156256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:29.156346 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.156371 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 12:26:29.156437 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 12:26:29.156509 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":34,"thread_start_us":110,"threads_started":1}
I20260806 12:26:29.156896 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.150244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.28.210.66:38743)
0806 12:26:29.150726 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:29.150738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:29.150762 (+ 24us) server_negotiation.cc:407] Connection header received
0806 12:26:29.150838 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:29.150844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:29.151051 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:29.151478 (+ 427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:29.151650 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.152868 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.154275 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.156020 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.156244 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.156402 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:29.156469 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 12:26:29.156568 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":168,"threads_started":1}
I20260806 12:26:29.157743 19092 tablet_bootstrap.cc:492] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e: No bootstrap required, opened a new log
I20260806 12:26:29.157866 19092 ts_tablet_manager.cc:1397] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 12:26:29.158387 19092 raft_consensus.cc:348] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:29.158545 19092 raft_consensus.cc:374] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:29.158596 19092 raft_consensus.cc:729] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c361ecb1cfb48ebb7a4738fb07a304e, State: Initialized, Role: FOLLOWER
I20260806 12:26:29.158737 19092 consensus_queue.cc:260] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:29.159019 19092 ts_tablet_manager.cc:1428] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 12:26:29.158820 19558 tablet_service.cc:1467] Processing CreateTablet for tablet eaed3fb47d954d7da720fffb711251a3 (DEFAULT_TABLE table=test-table [id=7f6a38081a084123b1c51cd830d0baf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:26:29.160286 19558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaed3fb47d954d7da720fffb711251a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:29.159857 19362 tablet_service.cc:1467] Processing CreateTablet for tablet eaed3fb47d954d7da720fffb711251a3 (DEFAULT_TABLE table=test-table [id=7f6a38081a084123b1c51cd830d0baf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:26:29.161017 19362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaed3fb47d954d7da720fffb711251a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:29.167789 19671 tablet_bootstrap.cc:492] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Bootstrap starting.
I20260806 12:26:29.168254 19672 tablet_bootstrap.cc:492] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: Bootstrap starting.
I20260806 12:26:29.170794 19672 tablet_bootstrap.cc:654] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:29.171705 19672 log.cc:824] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:29.173640 19671 tablet_bootstrap.cc:654] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:29.174044 19672 tablet_bootstrap.cc:492] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: No bootstrap required, opened a new log
I20260806 12:26:29.174248 19672 ts_tablet_manager.cc:1397] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260806 12:26:29.174475 19671 log.cc:824] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:29.176548 19671 tablet_bootstrap.cc:492] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: No bootstrap required, opened a new log
I20260806 12:26:29.176746 19671 ts_tablet_manager.cc:1397] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260806 12:26:29.178238 19672 raft_consensus.cc:348] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:29.178563 19672 raft_consensus.cc:374] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:29.178628 19672 raft_consensus.cc:729] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed7accba7534483aa9ddb1fadb2375b, State: Initialized, Role: FOLLOWER
I20260806 12:26:29.179131 19672 consensus_queue.cc:260] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:29.179934 19467 heartbeater.cc:503] Master 127.28.210.126:45431 was elected leader, sending a full tablet report...
I20260806 12:26:29.180294 19672 ts_tablet_manager.cc:1428] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260806 12:26:29.181599 19671 raft_consensus.cc:348] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:29.181816 19671 raft_consensus.cc:374] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:29.181882 19671 raft_consensus.cc:729] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9cfe34ee75e4b5eb3b1eacb7b67c7db, State: Initialized, Role: FOLLOWER
I20260806 12:26:29.182274 19671 consensus_queue.cc:260] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:29.182917 19653 heartbeater.cc:503] Master 127.28.210.126:45431 was elected leader, sending a full tablet report...
I20260806 12:26:29.183231 19671 ts_tablet_manager.cc:1428] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260806 12:26:29.212294 19252 tablet.cc:2369] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:26:29.333184 19257 raft_consensus.cc:482] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:26:29.333510 19257 raft_consensus.cc:504] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:29.334836 19257 leader_election.cc:288] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435), fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743)
I20260806 12:26:29.338950 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.335259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:40449 (local address 127.28.210.66:38743)
0806 12:26:29.335406 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:29.335411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:29.335621 (+ 210us) server_negotiation.cc:407] Connection header received
0806 12:26:29.335787 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:29.335791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:29.335847 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:29.335921 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:29.336279 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.336799 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.337960 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.338365 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.338474 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.338527 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:29.338755 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 12:26:29.338825 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 12:26:29.338950 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.335375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:54717 (local address 127.28.210.67:34435)
0806 12:26:29.335553 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:29.335558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:29.335731 (+ 173us) server_negotiation.cc:407] Connection header received
0806 12:26:29.335866 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:29.335868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:29.335914 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:29.335988 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:29.336276 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.336798 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.337957 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.338353 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.338460 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.338526 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:29.338738 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 12:26:29.338823 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 12:26:29.338959 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.335139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:38743 (local address 127.28.210.65:40449)
0806 12:26:29.335510 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:29.335526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:29.335659 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:29.335949 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:29.335952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:29.335985 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:29.336167 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:29.336174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.336915 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.336919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:29.337833 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:29.337843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.338482 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.338486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:29.338621 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.338656 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 12:26:29.338727 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 12:26:29.338805 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":102,"spinlock_wait_cycles":35328,"thread_start_us":93,"threads_started":1}
I20260806 12:26:29.338963 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.335139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:34435 (local address 127.28.210.65:54717)
0806 12:26:29.335629 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:29.335644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:29.335762 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:29.336010 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:29.336014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:29.336054 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:29.336165 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:29.336171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.336912 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.336916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:29.337837 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:29.337847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.338470 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.338474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:29.338598 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.338640 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 12:26:29.338701 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 12:26:29.338780 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":112,"spinlock_wait_cycles":768,"thread_start_us":72,"threads_started":1}
I20260806 12:26:29.339748 19393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaed3fb47d954d7da720fffb711251a3" candidate_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed7accba7534483aa9ddb1fadb2375b" is_pre_election: true
I20260806 12:26:29.340030 19591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaed3fb47d954d7da720fffb711251a3" candidate_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" is_pre_election: true
I20260806 12:26:29.340194 19393 raft_consensus.cc:2454] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c361ecb1cfb48ebb7a4738fb07a304e in term 0.
I20260806 12:26:29.340343 19591 raft_consensus.cc:2454] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c361ecb1cfb48ebb7a4738fb07a304e in term 0.
I20260806 12:26:29.340775 19084 leader_election.cc:302] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c361ecb1cfb48ebb7a4738fb07a304e, d9cfe34ee75e4b5eb3b1eacb7b67c7db, fed7accba7534483aa9ddb1fadb2375b; no voters:
I20260806 12:26:29.341060 19257 raft_consensus.cc:2781] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:26:29.341143 19257 raft_consensus.cc:482] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:26:29.341187 19257 raft_consensus.cc:3037] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:29.342303 19257 raft_consensus.cc:504] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:29.342669 19257 leader_election.cc:288] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 1 election: Requested vote from peers d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435), fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743)
I20260806 12:26:29.343076 19591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaed3fb47d954d7da720fffb711251a3" candidate_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db"
I20260806 12:26:29.343207 19591 raft_consensus.cc:3037] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:29.344218 19393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaed3fb47d954d7da720fffb711251a3" candidate_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed7accba7534483aa9ddb1fadb2375b"
I20260806 12:26:29.344339 19393 raft_consensus.cc:3037] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:29.344627 19591 raft_consensus.cc:2454] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c361ecb1cfb48ebb7a4738fb07a304e in term 1.
I20260806 12:26:29.345499 19393 raft_consensus.cc:2454] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c361ecb1cfb48ebb7a4738fb07a304e in term 1.
I20260806 12:26:29.345810 19083 leader_election.cc:302] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c361ecb1cfb48ebb7a4738fb07a304e, d9cfe34ee75e4b5eb3b1eacb7b67c7db, fed7accba7534483aa9ddb1fadb2375b; no voters:
I20260806 12:26:29.346025 19257 raft_consensus.cc:2781] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:26:29.346227 19257 raft_consensus.cc:686] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 LEADER]: Becoming Leader. State: Replica: 0c361ecb1cfb48ebb7a4738fb07a304e, State: Running, Role: LEADER
I20260806 12:26:29.346485 19257 consensus_queue.cc:237] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:29.349521 18956 catalog_manager.cc:5495] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c361ecb1cfb48ebb7a4738fb07a304e (127.28.210.65). New cstate: current_term: 1 leader_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } health_report { overall_health: HEALTHY } } }
W20260806 12:26:29.363624 19654 tablet.cc:2369] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:26:29.399574 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.395929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.28.210.65:38601)
0806 12:26:29.396223 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:29.396230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:29.396243 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:26:29.396323 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:29.396325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:29.396391 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:29.396499 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:26:29.396773 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.397373 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.398182 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:29.398438 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:29.398533 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:29.398739 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:26:29.399204 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:26:29.399306 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:29.399363 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 12:26:29.399420 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260806 12:26:29.403972 19393 raft_consensus.cc:1261] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b [term 1 FOLLOWER]: Refusing update from remote peer 0c361ecb1cfb48ebb7a4738fb07a304e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:26:29.403972 19591 raft_consensus.cc:1261] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 1 FOLLOWER]: Refusing update from remote peer 0c361ecb1cfb48ebb7a4738fb07a304e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:26:29.404860 19684 consensus_queue.cc:1035] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:26:29.405053 19257 consensus_queue.cc:1035] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:26:29.411700 19692 mvcc.cc:205] Tried to move back new op lower bound from 7315534599791063040 to 7315534599565934592. Current Snapshot: MvccSnapshot[applied={T|T < 7315534599791063040}]
I20260806 12:26:29.414325 19693 mvcc.cc:205] Tried to move back new op lower bound from 7315534599791063040 to 7315534599565934592. Current Snapshot: MvccSnapshot[applied={T|T < 7315534599791063040}]
I20260806 12:26:29.415375 19695 mvcc.cc:205] Tried to move back new op lower bound from 7315534599791063040 to 7315534599565934592. Current Snapshot: MvccSnapshot[applied={T|T < 7315534599791063040}]
I20260806 12:26:29.421267 18958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:26:29.421432 18958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:26:29.423851 18958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Sending DeleteTablet for 3 replicas of tablet eaed3fb47d954d7da720fffb711251a3
I20260806 12:26:29.424705 19362 tablet_service.cc:1514] Processing DeleteTablet for tablet eaed3fb47d954d7da720fffb711251a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:26:29 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:44218
I20260806 12:26:29.424675 19558 tablet_service.cc:1514] Processing DeleteTablet for tablet eaed3fb47d954d7da720fffb711251a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:26:29 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:51256
I20260806 12:26:29.424827 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 19017
I20260806 12:26:29.434523 19702 tablet_replica.cc:330] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: stopping tablet replica
I20260806 12:26:29.435148 19702 raft_consensus.cc:2229] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:26:29.435328 19703 tablet_replica.cc:330] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: stopping tablet replica
I20260806 12:26:29.435645 19703 raft_consensus.cc:2229] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:26:29.435781 19702 raft_consensus.cc:2258] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:26:29.435958 19703 raft_consensus.cc:2258] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:26:29.437599 19702 ts_tablet_manager.cc:1905] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:26:29.439947 19702 ts_tablet_manager.cc:1918] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:26:29.440141 19702 log.cc:1196] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/eaed3fb47d954d7da720fffb711251a3
I20260806 12:26:29.440493 19702 ts_tablet_manager.cc:1939] T eaed3fb47d954d7da720fffb711251a3 P d9cfe34ee75e4b5eb3b1eacb7b67c7db: Deleting consensus metadata
I20260806 12:26:29.441679 19703 ts_tablet_manager.cc:1905] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:26:29.441910 18937 catalog_manager.cc:4841] TS d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435): tablet eaed3fb47d954d7da720fffb711251a3 (table test-table [id=7f6a38081a084123b1c51cd830d0baf5]) successfully deleted
W20260806 12:26:29.444310 18933 connection.cc:535] client connection to 127.28.210.65:38601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 12:26:29.444490 18933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 12:26:29.444682 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 19255
I20260806 12:26:29.451352 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.451064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:38601 (local address 127.0.0.1:38628)
0806 12:26:29.451193 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:29.451269 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:38601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 12:26:29.451828 18933 catalog_manager.cc:4584] TS 0c361ecb1cfb48ebb7a4738fb07a304e (127.28.210.65:38601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eaed3fb47d954d7da720fffb711251a3: Network error: Client connection negotiation failed: client connection to 127.28.210.65:38601: connect: Connection refused (error 111)
I20260806 12:26:29.458642 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 19471
I20260806 12:26:29.462647 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.458768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:38743 (local address 127.0.0.1:44228)
0806 12:26:29.462473 (+ 3705us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:29.462546 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.66:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3588}
I20260806 12:26:29.478356 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 18907
I20260806 12:26:29.493914 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:45431
--webserver_interface=127.28.210.126
--webserver_port=37141
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:45431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:29.610845 19705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:29.611121 19705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:29.611164 19705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:29.615264 19705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:26:29.615340 19705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:29.615360 19705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:29.615375 19705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:26:29.615389 19705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:26:29.620824 19705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:45431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:45431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:29.621951 19705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:29.622994 19705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:29.631524 19711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:29.631690 19705 server_base.cc:1034] running on GCE node
W20260806 12:26:29.631546 19710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:29.631531 19713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:29.632522 19705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:29.633692 19705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:29.634856 19705 hybrid_clock.cc:648] HybridClock initialized: now 1786019189634837 us; error 35 us; skew 500 ppm
Aug 06 12:26:29 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019189, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:29.637766 19705 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:26:29.638808 19705 webserver.cc:460] Webserver started at http://127.28.210.126:37141/ using document root <none> and password file <none>
I20260806 12:26:29.639429 19705 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:29.639506 19705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:29.644058 19705 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:29.646656 19726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:29.648039 19705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 12:26:29.648216 19705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "0e231c1feced4d6491aea9110bbf86ad"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "1f2c440212b04359a87dfcaf6da1214f"
server_key_iv: "dab0aeef9ec6f74d698d4ed9e924cd7d"
server_key_version: "encryptionkey@0"
I20260806 12:26:29.648873 19705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:29.669211 19705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:29.672587 19705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:29.672780 19705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:29.679953 19705 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:45431
I20260806 12:26:29.680253 19804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:45431 every 8 connection(s)
I20260806 12:26:29.681212 19705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:26:29.681493 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 19705
I20260806 12:26:29.681946 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:38601
--local_ip_for_outbound_sockets=127.28.210.65
--tserver_master_addrs=127.28.210.126:45431
--webserver_port=34321
--webserver_interface=127.28.210.65
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:26:29.686131 19806 sys_catalog.cc:263] Verifying existing consensus state
I20260806 12:26:29.687959 19806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Bootstrap starting.
I20260806 12:26:29.706228 19806 log.cc:824] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:29.717538 19806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 12:26:29.717898 19806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad: Bootstrap complete.
I20260806 12:26:29.721932 19806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } }
I20260806 12:26:29.722332 19806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e231c1feced4d6491aea9110bbf86ad, State: Initialized, Role: FOLLOWER
I20260806 12:26:29.722796 19806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } }
I20260806 12:26:29.722923 19806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:26:29.723002 19806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:26:29.723097 19806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 2 FOLLOWER]: Advancing to term 3
I20260806 12:26:29.725072 19806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } }
I20260806 12:26:29.725391 19806 leader_election.cc:302] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e231c1feced4d6491aea9110bbf86ad; no voters:
I20260806 12:26:29.725690 19806 leader_election.cc:288] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 12:26:29.725796 19812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 3 FOLLOWER]: Leader election won for term 3
I20260806 12:26:29.726015 19812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [term 3 LEADER]: Becoming Leader. State: Replica: 0e231c1feced4d6491aea9110bbf86ad, State: Running, Role: LEADER
I20260806 12:26:29.726317 19812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } }
I20260806 12:26:29.726999 19806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:26:29.727717 19813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0e231c1feced4d6491aea9110bbf86ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } } }
I20260806 12:26:29.727843 19813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:29.728469 19822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:26:29.728508 19815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e231c1feced4d6491aea9110bbf86ad. Latest consensus state: current_term: 3 leader_uuid: "0e231c1feced4d6491aea9110bbf86ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e231c1feced4d6491aea9110bbf86ad" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 45431 } } }
I20260806 12:26:29.728621 19815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e231c1feced4d6491aea9110bbf86ad [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:29.733901 19822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:26:29.734655 19822 catalog_manager.cc:1259] Loaded cluster ID: c53e24f756ae4023bde097827e06fb3a
I20260806 12:26:29.734723 19822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 12:26:29.735847 19822 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8d4be229ad google::LogMessage::Fail() at ??:0
@ 0x7f8d4be2588a google::LogMessage::SendToLog() at ??:0
@ 0x7f8d4be224de google::LogMessage::Flush() at ??:0
@ 0x7f8d4be23a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8d56761fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8d56762d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8d5675f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8d567a3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8d4e0743d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8d4e074c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f8d4e0763f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8d4e065512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8d4f6a96db start_thread at ??:0
@ 0x7f8d4a92061f clone at ??:0
W20260806 12:26:29.831903 19809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:29.832245 19809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:29.832300 19809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:29.836548 19809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:29.836613 19809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:29.836678 19809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:26:29.842306 19809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:38601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=34321
--tserver_master_addrs=127.28.210.126:45431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:29.843477 19809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:29.844587 19809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:29.854190 19867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:29.854252 19864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:29.854234 19863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:29.854535 19809 server_base.cc:1034] running on GCE node
I20260806 12:26:29.854938 19809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:29.855618 19809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:29.856794 19809 hybrid_clock.cc:648] HybridClock initialized: now 1786019189856775 us; error 35 us; skew 500 ppm
Aug 06 12:26:29 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019189, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:29.860000 19809 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:26:29.861230 19809 webserver.cc:460] Webserver started at http://127.28.210.65:34321/ using document root <none> and password file <none>
I20260806 12:26:29.861881 19809 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:29.861961 19809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:29.866529 19809 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:26:29.868952 19877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:29.870420 19809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:29.870546 19809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "0c361ecb1cfb48ebb7a4738fb07a304e"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "21f807b8fc3aee6c9b221f9a52478622"
server_key_iv: "731c1e35d04c2e379ca482beca85588e"
server_key_version: "encryptionkey@0"
I20260806 12:26:29.871150 19809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:29.894256 19809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:29.898363 19809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:29.898618 19809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:29.899335 19809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:29.900686 19888 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 12:26:29.906596 19809 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 12:26:29.906672 19809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 12:26:29.906759 19809 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 12:26:29.908655 19809 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 12:26:29.908706 19809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260806 12:26:29.909060 19888 tablet_bootstrap.cc:492] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Bootstrap starting.
I20260806 12:26:29.918669 19809 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:38601
I20260806 12:26:29.918941 20045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:38601 every 8 connection(s)
I20260806 12:26:29.919731 19809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:26:29.920675 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 19809
I20260806 12:26:29.921165 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:38743
--local_ip_for_outbound_sockets=127.28.210.66
--tserver_master_addrs=127.28.210.126:45431
--webserver_port=34893
--webserver_interface=127.28.210.66
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:26:29.932121 19888 log.cc:824] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:29.942881 19888 tablet_bootstrap.cc:492] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 12:26:29.943404 19888 tablet_bootstrap.cc:492] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Bootstrap complete.
I20260806 12:26:29.944979 19888 ts_tablet_manager.cc:1397] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Time spent bootstrapping tablet: real 0.036s user 0.018s sys 0.016s
I20260806 12:26:29.948537 19888 raft_consensus.cc:348] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:29.949136 19888 raft_consensus.cc:729] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c361ecb1cfb48ebb7a4738fb07a304e, State: Initialized, Role: FOLLOWER
I20260806 12:26:29.949723 19888 consensus_queue.cc:260] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:29.950917 19888 ts_tablet_manager.cc:1428] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260806 12:26:29.972653 20051 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:26:29.925143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:53485)
0806 12:26:29.925774 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:29.925840 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:29.926900 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:29.971375 (+ 44475us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":178,"threads_started":1}
W20260806 12:26:29.973598 20046 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:45431 (0 consecutive failures): Network error: Failed to ping master at 127.28.210.126:45431: Client connection negotiation failed: client connection to 127.28.210.126:45431: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 12:26:29.974546 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.974189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:35149)
0806 12:26:29.974375 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:29.974467 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 12:26:29.975961 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:29.975632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:43221)
0806 12:26:29.975834 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:29.975888 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260806 12:26:29.976197 20046 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 12:26:30.047911 20049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:30.048249 20049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:30.048300 20049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:30.052407 20049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:30.052469 20049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:30.052531 20049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:26:30.058012 20049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:38743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=34893
--tserver_master_addrs=127.28.210.126:45431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:30.059207 20049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:30.060405 20049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:26:30.071297 20049 server_base.cc:1034] running on GCE node
W20260806 12:26:30.071235 20061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:30.071238 20066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:30.071238 20064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:30.074081 20049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:30.074995 20049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:30.076191 20049 hybrid_clock.cc:648] HybridClock initialized: now 1786019190076166 us; error 46 us; skew 500 ppm
Aug 06 12:26:30 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019190, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:30.079249 20049 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:26:30.080349 20049 webserver.cc:460] Webserver started at http://127.28.210.66:34893/ using document root <none> and password file <none>
I20260806 12:26:30.080955 20049 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:30.081035 20049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:30.086221 20049 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:26:30.089113 20073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:30.090553 20049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:30.090694 20049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "fed7accba7534483aa9ddb1fadb2375b"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "c35618e9830cb0e4ddfd900ae7e325d3"
server_key_iv: "2706f7073b78662f39b9deb6a7788f10"
server_key_version: "encryptionkey@0"
I20260806 12:26:30.091318 20049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:30.112965 20049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:30.116250 20049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:30.116439 20049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:30.117100 20049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:30.118297 20081 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260806 12:26:30.123051 20081 ts_tablet_manager.cc:1852] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260806 12:26:30.123579 20081 ts_tablet_manager.cc:1905] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:26:30.128713 20081 ts_tablet_manager.cc:1918] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:26:30.128854 20081 log.cc:1196] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/eaed3fb47d954d7da720fffb711251a3
I20260806 12:26:30.129235 20081 ts_tablet_manager.cc:1939] T eaed3fb47d954d7da720fffb711251a3 P fed7accba7534483aa9ddb1fadb2375b: Deleting consensus metadata
I20260806 12:26:30.129587 20049 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260806 12:26:30.129683 20049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.001s sys 0.000s
I20260806 12:26:30.129778 20049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:30.129819 20049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:30.141309 20049 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:38743
I20260806 12:26:30.141351 20246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:38743 every 8 connection(s)
I20260806 12:26:30.142489 20049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:26:30.145188 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 20049
I20260806 12:26:30.145642 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:34435
--local_ip_for_outbound_sockets=127.28.210.67
--tserver_master_addrs=127.28.210.126:45431
--webserver_port=37115
--webserver_interface=127.28.210.67
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:26:30.146781 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:30.145130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:46117)
0806 12:26:30.145729 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:30.145884 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":150,"threads_started":1}
W20260806 12:26:30.147513 20247 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:45431 (0 consecutive failures): Network error: Failed to ping master at 127.28.210.126:45431: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
I20260806 12:26:30.148531 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:30.148186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:36041)
0806 12:26:30.148333 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:30.148405 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 12:26:30.149416 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:30.149142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:56819)
0806 12:26:30.149275 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:30.149338 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 12:26:30.149744 20247 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 12:26:30.265525 20251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:30.265818 20251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:30.265883 20251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:30.270257 20251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:30.270334 20251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:30.270421 20251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:26:30.275993 20251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:34435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=37115
--tserver_master_addrs=127.28.210.126:45431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:30.277173 20251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:30.278231 20251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:30.288946 20263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:30.288936 20258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:30.289000 20251 server_base.cc:1034] running on GCE node
W20260806 12:26:30.288937 20261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:30.289743 20251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:30.290550 20251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:30.291754 20251 hybrid_clock.cc:648] HybridClock initialized: now 1786019190291732 us; error 32 us; skew 500 ppm
Aug 06 12:26:30 dist-test-slave-lgr5 krb5kdc[18121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019190, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:30.294914 20251 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:26:30.295939 20251 webserver.cc:460] Webserver started at http://127.28.210.67:37115/ using document root <none> and password file <none>
I20260806 12:26:30.296571 20251 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:30.296643 20251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:30.300806 20251 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:30.303133 20274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:30.304579 20251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:30.304702 20251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db"
format_stamp: "Formatted at 2026-08-06 12:26:27 on dist-test-slave-lgr5"
server_key: "558843e5dfa681769ace9339cda1b0af"
server_key_iv: "f5a4706e3f6504cb7582531d89bbbd0f"
server_key_version: "encryptionkey@0"
I20260806 12:26:30.305294 20251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:30.318189 20251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:30.321583 20251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:30.321781 20251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:30.322445 20251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:30.323659 20251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:30.323714 20251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:30.323757 20251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:30.323777 20251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:30.333370 20251 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:34435
I20260806 12:26:30.333446 20438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:34435 every 8 connection(s)
I20260806 12:26:30.334467 20251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:26:30.335134 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 20251
I20260806 12:26:30.338570 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:30.336865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:42147)
0806 12:26:30.337392 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:30.337834 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":124,"threads_started":1}
W20260806 12:26:30.339046 20440 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:45431 (0 consecutive failures): Network error: Failed to ping master at 127.28.210.126:45431: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
I20260806 12:26:30.339679 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:30.339435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:56945)
0806 12:26:30.339572 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:30.339621 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 12:26:30.340265 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:30.340061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:58401)
0806 12:26:30.340174 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:30.340215 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 12:26:30.340426 20440 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 12:26:30.977372 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:30.976853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:59285)
0806 12:26:30.977179 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:30.977248 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":94,"threads_started":1}
I20260806 12:26:31.106070 20445 raft_consensus.cc:482] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:26:31.106281 20445 raft_consensus.cc:504] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:31.107386 20445 leader_election.cc:288] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435), fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743)
Aug 06 12:26:31 dist-test-slave-lgr5 krb5kdc[18121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019189, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.66@KRBTEST.COM
Aug 06 12:26:31 dist-test-slave-lgr5 krb5kdc[18121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019189, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.67@KRBTEST.COM
I20260806 12:26:31.125654 20449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:31.107722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:34435 (local address 127.28.210.65:43427)
0806 12:26:31.108445 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:31.108457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:31.108553 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:31.110267 (+ 1714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:31.110272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:31.110856 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:31.111631 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:31.111659 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:31.113910 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:31.113915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:31.115662 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:31.115682 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:31.116895 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:31.116899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:31.117256 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:31.118058 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:31.118089 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:31.120453 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:31.122620 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:31.122623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:31.122624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:31.123008 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:31.123356 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:31.123359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:31.123361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:31.123521 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:31.124436 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:31.124444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:31.124784 (+ 340us) client_negotiation.cc:769] Sending connection context
0806 12:26:31.125032 (+ 248us) client_negotiation.cc:240] Negotiation successful
0806 12:26:31.125420 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":694,"spinlock_wait_cycles":18688,"thread_start_us":64,"threads_started":1}
I20260806 12:26:31.125655 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:31.107721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:38743 (local address 127.28.210.65:33921)
0806 12:26:31.107895 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:31.107909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:31.108014 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:31.110265 (+ 2251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:31.110272 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:31.110846 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:31.111631 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:31.111659 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:31.113910 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:31.113915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:31.115662 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:31.115682 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:31.116895 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:31.116899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:31.117261 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:31.118058 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:31.118089 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:31.120221 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:31.122567 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:31.122575 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:31.122592 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:31.123008 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:31.123369 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:31.123373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:31.123374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:31.123536 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:31.124436 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:31.124444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:31.124784 (+ 340us) client_negotiation.cc:769] Sending connection context
0806 12:26:31.125032 (+ 248us) client_negotiation.cc:240] Negotiation successful
0806 12:26:31.125516 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":537,"spinlock_wait_cycles":88320}
I20260806 12:26:31.125738 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:31.107863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:33921 (local address 127.28.210.66:38743)
0806 12:26:31.108314 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:31.108330 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:31.108377 (+ 47us) server_negotiation.cc:407] Connection header received
0806 12:26:31.109300 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:31.109310 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:31.109810 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:31.110492 (+ 682us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:31.111904 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:31.113649 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:31.115864 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:31.116717 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:31.116935 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:31.120554 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:31.120564 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:31.120585 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:31.120624 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:31.122398 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:31.123129 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:31.123134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:31.123139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:31.123262 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:31.123647 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:31.123650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:31.123652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:31.124247 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:31.124374 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:31.125211 (+ 837us) server_negotiation.cc:299] Negotiation successful
0806 12:26:31.125546 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":107,"threads_started":1}
I20260806 12:26:31.125902 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:31.107863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:43427 (local address 127.28.210.67:34435)
0806 12:26:31.108314 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:31.108330 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:31.108552 (+ 222us) server_negotiation.cc:407] Connection header received
0806 12:26:31.109327 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:31.109334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:31.109811 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:31.110172 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:31.111904 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:31.113649 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:31.115864 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:31.116771 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:31.116921 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:31.120671 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:31.120676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:31.120688 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:31.120721 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:31.122499 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:31.123129 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:31.123134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:31.123139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:31.123261 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:31.123616 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:31.123619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:31.123621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:31.124247 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:31.124374 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:31.125211 (+ 837us) server_negotiation.cc:299] Negotiation successful
0806 12:26:31.125546 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":107,"threads_started":1}
W20260806 12:26:31.128763 19882 leader_election.cc:341] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743): Not found: Tablet not found: eaed3fb47d954d7da720fffb711251a3
W20260806 12:26:31.129241 19883 leader_election.cc:341] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435): Not found: Tablet not found: eaed3fb47d954d7da720fffb711251a3
I20260806 12:26:31.129370 19883 leader_election.cc:302] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c361ecb1cfb48ebb7a4738fb07a304e; no voters: d9cfe34ee75e4b5eb3b1eacb7b67c7db, fed7accba7534483aa9ddb1fadb2375b
I20260806 12:26:31.129657 20445 raft_consensus.cc:2726] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 12:26:31.150926 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:31.150440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:39613)
0806 12:26:31.150762 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:31.150830 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":79,"threads_started":1}
I20260806 12:26:31.341645 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:31.341125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:48399)
0806 12:26:31.341431 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:31.341505 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":76,"threads_started":1}
I20260806 12:26:31.978719 20452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:31.978170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:55489)
0806 12:26:31.978505 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:31.978586 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":99,"threads_started":1}
I20260806 12:26:32.152315 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:32.151796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:48277)
0806 12:26:32.152127 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:32.152197 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":106,"threads_started":1}
I20260806 12:26:32.342981 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:32.342488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:41653)
0806 12:26:32.342775 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:32.342849 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":82,"threads_started":1}
I20260806 12:26:32.673552 20455 raft_consensus.cc:482] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:26:32.673709 20455 raft_consensus.cc:504] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:32.674151 20455 leader_election.cc:288] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435), fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743)
W20260806 12:26:32.674996 19883 leader_election.cc:341] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435): Not found: Tablet not found: eaed3fb47d954d7da720fffb711251a3
W20260806 12:26:32.675205 19882 leader_election.cc:341] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743): Not found: Tablet not found: eaed3fb47d954d7da720fffb711251a3
I20260806 12:26:32.675307 19882 leader_election.cc:302] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c361ecb1cfb48ebb7a4738fb07a304e; no voters: d9cfe34ee75e4b5eb3b1eacb7b67c7db, fed7accba7534483aa9ddb1fadb2375b
I20260806 12:26:32.675480 20455 raft_consensus.cc:2726] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 12:26:32.980412 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:32.979732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:50195)
0806 12:26:32.980138 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:32.980219 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":97,"threads_started":1}
I20260806 12:26:33.153687 20457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:33.153219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:34771)
0806 12:26:33.153526 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:33.153588 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":100,"threads_started":1}
I20260806 12:26:33.344347 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:33.343828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:54541)
0806 12:26:33.344172 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:33.344241 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
I20260806 12:26:33.981740 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:33.981198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:52047)
0806 12:26:33.981574 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:33.981632 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":95,"threads_started":1}
I20260806 12:26:34.155238 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:34.154631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:50449)
0806 12:26:34.154997 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:34.155101 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":88,"threads_started":1}
I20260806 12:26:34.345778 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:34.345275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:52581)
0806 12:26:34.345603 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:34.345671 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":126,"threads_started":1}
I20260806 12:26:34.944939 20465 raft_consensus.cc:482] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:26:34.945086 20465 raft_consensus.cc:504] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:34.945549 20465 leader_election.cc:288] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435), fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743)
W20260806 12:26:34.946309 19882 leader_election.cc:341] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743): Not found: Tablet not found: eaed3fb47d954d7da720fffb711251a3
W20260806 12:26:34.946444 19883 leader_election.cc:341] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435): Not found: Tablet not found: eaed3fb47d954d7da720fffb711251a3
I20260806 12:26:34.946534 19883 leader_election.cc:302] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c361ecb1cfb48ebb7a4738fb07a304e; no voters: d9cfe34ee75e4b5eb3b1eacb7b67c7db, fed7accba7534483aa9ddb1fadb2375b
I20260806 12:26:34.946700 20465 raft_consensus.cc:2726] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 12:26:34.983286 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:34.982753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:56463)
0806 12:26:34.983115 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:34.983181 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":80,"threads_started":1}
I20260806 12:26:35.156937 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:35.156350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:46721)
0806 12:26:35.156727 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:35.156804 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":121,"threads_started":1}
I20260806 12:26:35.347078 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:35.346600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:41707)
0806 12:26:35.346908 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:35.346977 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":95,"threads_started":1}
I20260806 12:26:35.984647 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:35.984175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:49653)
0806 12:26:35.984471 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:35.984538 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260806 12:26:36.158537 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:36.157986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:53019)
0806 12:26:36.158338 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:36.158422 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":83,"threads_started":1}
I20260806 12:26:36.348690 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:36.348117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:44685)
0806 12:26:36.348512 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:36.348583 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":106,"threads_started":1}
I20260806 12:26:36.986073 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:36.985592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:39283)
0806 12:26:36.985884 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:36.985965 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260806 12:26:37.128645 20477 raft_consensus.cc:482] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:26:37.128794 20477 raft_consensus.cc:504] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:37.129289 20477 leader_election.cc:288] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435), fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743)
W20260806 12:26:37.130179 19883 leader_election.cc:341] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435): Not found: Tablet not found: eaed3fb47d954d7da720fffb711251a3
W20260806 12:26:37.130291 19882 leader_election.cc:341] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743): Not found: Tablet not found: eaed3fb47d954d7da720fffb711251a3
I20260806 12:26:37.130389 19882 leader_election.cc:302] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c361ecb1cfb48ebb7a4738fb07a304e; no voters: d9cfe34ee75e4b5eb3b1eacb7b67c7db, fed7accba7534483aa9ddb1fadb2375b
I20260806 12:26:37.130555 20477 raft_consensus.cc:2726] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 12:26:37.159811 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:37.159322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:56421)
0806 12:26:37.159639 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:37.159698 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":77,"threads_started":1}
I20260806 12:26:37.350006 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:37.349526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:55775)
0806 12:26:37.349849 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:37.349906 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":76,"threads_started":1}
I20260806 12:26:37.987409 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:37.986941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:57335)
0806 12:26:37.987231 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:37.987295 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
I20260806 12:26:38.161283 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:38.160761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:55305)
0806 12:26:38.161103 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:38.161171 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":93,"threads_started":1}
I20260806 12:26:38.351351 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:38.350862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:35141)
0806 12:26:38.351174 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:38.351245 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":87,"threads_started":1}
I20260806 12:26:38.988854 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:38.988350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:45733)
0806 12:26:38.988667 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:38.988747 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
I20260806 12:26:39.162899 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:39.162352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:50793)
0806 12:26:39.162686 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:39.162753 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":90,"threads_started":1}
I20260806 12:26:39.352769 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:39.352248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:33933)
0806 12:26:39.352592 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:39.352659 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260806 12:26:39.737576 20488 raft_consensus.cc:482] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:26:39.737739 20488 raft_consensus.cc:504] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cfe34ee75e4b5eb3b1eacb7b67c7db" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 34435 } } peers { permanent_uuid: "fed7accba7534483aa9ddb1fadb2375b" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 38743 } } peers { permanent_uuid: "0c361ecb1cfb48ebb7a4738fb07a304e" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 38601 } }
I20260806 12:26:39.738237 20488 leader_election.cc:288] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435), fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743)
W20260806 12:26:39.739113 19883 leader_election.cc:341] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9cfe34ee75e4b5eb3b1eacb7b67c7db (127.28.210.67:34435): Not found: Tablet not found: eaed3fb47d954d7da720fffb711251a3
W20260806 12:26:39.739244 19882 leader_election.cc:341] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fed7accba7534483aa9ddb1fadb2375b (127.28.210.66:38743): Not found: Tablet not found: eaed3fb47d954d7da720fffb711251a3
I20260806 12:26:39.739341 19882 leader_election.cc:302] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c361ecb1cfb48ebb7a4738fb07a304e; no voters: d9cfe34ee75e4b5eb3b1eacb7b67c7db, fed7accba7534483aa9ddb1fadb2375b
I20260806 12:26:39.739578 20488 raft_consensus.cc:2726] T eaed3fb47d954d7da720fffb711251a3 P 0c361ecb1cfb48ebb7a4738fb07a304e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 12:26:39.990229 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:39.989739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:56185)
0806 12:26:39.990053 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:39.990115 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
I20260806 12:26:40.164302 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:40.163779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:52199)
0806 12:26:40.164126 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:40.164198 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
I20260806 12:26:40.354115 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:40.353634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:60221)
0806 12:26:40.353935 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:40.354005 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":105,"threads_started":1}
I20260806 12:26:40.991761 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:40.991259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:57057)
0806 12:26:40.991590 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:40.991657 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
I20260806 12:26:41.165570 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:41.165067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:35393)
0806 12:26:41.165394 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:41.165459 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":96,"threads_started":1}
I20260806 12:26:41.355531 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:41.355027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:38409)
0806 12:26:41.355350 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:41.355419 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":108,"threads_started":1}
I20260806 12:26:41.993309 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:41.992761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:32841)
0806 12:26:41.993096 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:41.993169 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":92,"threads_started":1}
I20260806 12:26:42.166904 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:42.166463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:39329)
0806 12:26:42.166732 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:42.166798 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260806 12:26:42.356894 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:42.356437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:53701)
0806 12:26:42.356722 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:42.356788 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260806 12:26:42.994793 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:42.994236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:50811)
0806 12:26:42.994562 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:42.994664 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":97,"threads_started":1}
I20260806 12:26:43.168332 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:43.167772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:58097)
0806 12:26:43.168166 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:43.168227 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":128,"threads_started":1}
I20260806 12:26:43.358161 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:43.357646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:33943)
0806 12:26:43.357988 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:43.358052 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":83,"threads_started":1}
I20260806 12:26:43.996145 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:43.995637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:57557)
0806 12:26:43.995935 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:43.995999 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":108,"threads_started":1}
I20260806 12:26:44.169785 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:44.169259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:36473)
0806 12:26:44.169593 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:44.169665 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":88,"threads_started":1}
I20260806 12:26:44.359555 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:44.359086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:43683)
0806 12:26:44.359381 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:44.359443 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
I20260806 12:26:44.997720 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:44.997252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.65:60485)
0806 12:26:44.997544 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:44.997614 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260806 12:26:45.171257 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:45.170799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.66:50219)
0806 12:26:45.171088 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:45.171151 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
I20260806 12:26:45.344619 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 19809
I20260806 12:26:45.360683 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 20049
I20260806 12:26:45.360980 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:45.360550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:45431 (local address 127.28.210.67:45835)
0806 12:26:45.360815 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:45.360874 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260806 12:26:45.374125 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 20251
2026-08-06T12:26:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19728 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:26:45 dist-test-slave-lgr5 krb5kdc[20514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:26:45 dist-test-slave-lgr5 krb5kdc[20514](info): set up 2 sockets
Aug 06 12:26:45 dist-test-slave-lgr5 krb5kdc[20514](info): commencing operation
krb5kdc: starting...
W20260806 12:26:46.546295 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:26:46 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:26:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:26:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:26:46.680559 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34979
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:34979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:46.800441 20529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:46.800743 20529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:46.800786 20529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:46.804874 20529 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 12:26:46.804935 20529 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 12:26:46.804950 20529 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 12:26:46.804963 20529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:26:46.804983 20529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:46.805017 20529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:46.805044 20529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:26:46.805070 20529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:26:46.810732 20529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:34979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:46.811966 20529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:46.813094 20529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:46.822013 20538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:46.821997 20537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:46.822140 20529 server_base.cc:1034] running on GCE node
W20260806 12:26:46.821997 20544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:46.822929 20529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:46.824160 20529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:46.825317 20529 hybrid_clock.cc:648] HybridClock initialized: now 1786019206825299 us; error 40 us; skew 500 ppm
Aug 06 12:26:46 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019206, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:46.828061 20529 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:26:46.829052 20529 webserver.cc:460] Webserver started at http://127.28.210.126:45061/ using document root <none> and password file <none>
I20260806 12:26:46.829600 20529 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:46.829658 20529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:46.829833 20529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:26:46.832060 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "5cefe36342cc44e387733f3108482125"
format_stamp: "Formatted at 2026-08-06 12:26:46 on dist-test-slave-lgr5"
server_key: "535fc38864e16fca80c83a6d6a2a2aa1"
server_key_iv: "757c9d8208d0095e51afac08bcae7e30"
server_key_version: "encryptionkey@0"
I20260806 12:26:46.832711 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "5cefe36342cc44e387733f3108482125"
format_stamp: "Formatted at 2026-08-06 12:26:46 on dist-test-slave-lgr5"
server_key: "535fc38864e16fca80c83a6d6a2a2aa1"
server_key_iv: "757c9d8208d0095e51afac08bcae7e30"
server_key_version: "encryptionkey@0"
I20260806 12:26:46.836880 20529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 12:26:46.839614 20553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:46.840992 20529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 12:26:46.841130 20529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "5cefe36342cc44e387733f3108482125"
format_stamp: "Formatted at 2026-08-06 12:26:46 on dist-test-slave-lgr5"
server_key: "535fc38864e16fca80c83a6d6a2a2aa1"
server_key_iv: "757c9d8208d0095e51afac08bcae7e30"
server_key_version: "encryptionkey@0"
I20260806 12:26:46.841245 20529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:46.882099 20529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:46.885329 20529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:46.885533 20529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:46.892539 20529 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:34979
I20260806 12:26:46.892812 20627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:34979 every 8 connection(s)
I20260806 12:26:46.893556 20529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:26:46.896575 20628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:46.898722 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 20529
I20260806 12:26:46.898874 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:26:46.899197 29513 external_mini_cluster.cc:1428] Setting key 7975e9a24ecb45e0aae210474000008b
I20260806 12:26:46.903633 20628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Bootstrap starting.
I20260806 12:26:46.906013 20628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:46.906914 20628 log.cc:824] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Log is configured to *not* fsync() on all Append() calls
Aug 06 12:26:46 dist-test-slave-lgr5 krb5kdc[20514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:46.909204 20628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: No bootstrap required, opened a new log
I20260806 12:26:46.914003 20628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } }
I20260806 12:26:46.914233 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:46.900713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.28.210.126:34979)
0806 12:26:46.901239 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:46.901256 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:46.901334 (+ 78us) server_negotiation.cc:407] Connection header received
0806 12:26:46.902050 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:46.902068 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:46.902407 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:46.902737 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:46.903630 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:46.904727 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:46.905612 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:46.906095 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:46.906277 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:46.908973 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:46.908980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:46.908999 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:46.909031 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:46.910919 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:46.911394 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:46.911400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:46.911406 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:46.911565 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:46.911804 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:46.911807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:46.911809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:46.912410 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:46.912609 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:46.912864 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 12:26:46.913204 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":157,"threads_started":1}
I20260806 12:26:46.914361 20628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:46.914626 20628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cefe36342cc44e387733f3108482125, State: Initialized, Role: FOLLOWER
I20260806 12:26:46.915165 20628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } }
I20260806 12:26:46.915349 20628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:26:46.915416 20628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:26:46.915521 20628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:46.916615 20628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } }
I20260806 12:26:46.916970 20628 leader_election.cc:302] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cefe36342cc44e387733f3108482125; no voters:
I20260806 12:26:46.917308 20628 leader_election.cc:288] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:26:46.917425 20637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:26:46.917659 20637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 1 LEADER]: Becoming Leader. State: Replica: 5cefe36342cc44e387733f3108482125, State: Running, Role: LEADER
I20260806 12:26:46.918012 20637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } }
I20260806 12:26:46.918567 20628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:26:46.919728 20640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cefe36342cc44e387733f3108482125. Latest consensus state: current_term: 1 leader_uuid: "5cefe36342cc44e387733f3108482125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } } }
I20260806 12:26:46.919735 20639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cefe36342cc44e387733f3108482125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } } }
I20260806 12:26:46.919909 20640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: This master's current role is: LEADER
I20260806 12:26:46.919953 20639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: This master's current role is: LEADER
W20260806 12:26:46.923620 20655 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:26:46.923712 20655 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:26:46.923796 20647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:26:46.924572 20647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:26:46.931236 20647 catalog_manager.cc:1347] Generated new cluster ID: 577593c98d6445ba9e0a9ce7778398e1
I20260806 12:26:46.931337 20647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:26:46.952566 20647 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:26:46.953868 20647 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:26:46.974011 20647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Generated new TSK 0
I20260806 12:26:46.974951 20647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:26:47.020218 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34979
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:47.140065 20662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:47.140417 20662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:47.140466 20662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:47.144738 20662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:47.144816 20662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:47.144897 20662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:26:47.150651 20662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:47.151777 20662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:47.152854 20662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:47.163139 20669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:47.163139 20673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:47.163158 20668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:47.163410 20662 server_base.cc:1034] running on GCE node
I20260806 12:26:47.164103 20662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:47.164850 20662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:47.166033 20662 hybrid_clock.cc:648] HybridClock initialized: now 1786019207166000 us; error 45 us; skew 500 ppm
Aug 06 12:26:47 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019207, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:47.169100 20662 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:26:47.170158 20662 webserver.cc:460] Webserver started at http://127.28.210.65:33697/ using document root <none> and password file <none>
I20260806 12:26:47.170770 20662 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:47.170843 20662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:47.171051 20662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:26:47.173386 20662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "fc8ef18fff6b43e6b2422e21212148a3"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "5a24654cf2364cf6a88a39b77ab94679"
server_key_iv: "405768457ada9e32ac963f66507d5140"
server_key_version: "encryptionkey@0"
I20260806 12:26:47.174081 20662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "fc8ef18fff6b43e6b2422e21212148a3"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "5a24654cf2364cf6a88a39b77ab94679"
server_key_iv: "405768457ada9e32ac963f66507d5140"
server_key_version: "encryptionkey@0"
I20260806 12:26:47.178346 20662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:26:47.181278 20681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:47.182768 20662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:26:47.182914 20662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "fc8ef18fff6b43e6b2422e21212148a3"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "5a24654cf2364cf6a88a39b77ab94679"
server_key_iv: "405768457ada9e32ac963f66507d5140"
server_key_version: "encryptionkey@0"
I20260806 12:26:47.183043 20662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:47.197687 20662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:47.201159 20662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:47.201368 20662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:47.202026 20662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:47.202983 20662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:47.203033 20662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:47.203090 20662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:47.203118 20662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:47.212430 20662 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:34221
I20260806 12:26:47.212458 20845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:34221 every 8 connection(s)
I20260806 12:26:47.213585 20662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:26:47.218820 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 20662
I20260806 12:26:47.218966 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:26:47.219295 29513 external_mini_cluster.cc:1428] Setting key 700e4f66d81c66dc82a0139d50936c53
Aug 06 12:26:47 dist-test-slave-lgr5 krb5kdc[20514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019207, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:47.230371 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.215839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:60035 (local address 127.28.210.126:34979)
0806 12:26:47.216031 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:47.216035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:47.217100 (+ 1065us) server_negotiation.cc:407] Connection header received
0806 12:26:47.218582 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:47.218586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:47.218653 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:47.218735 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:47.220349 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.220895 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.222166 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.222407 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.222511 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.225264 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:47.225268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:47.225270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:47.225299 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:47.226782 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:47.227570 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:47.227574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:47.227576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:47.227716 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:47.228796 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:47.228799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:47.228801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:47.229089 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:47.229204 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:47.230009 (+ 805us) server_negotiation.cc:299] Negotiation successful
0806 12:26:47.230185 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:26:47.231424 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.216275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:60035)
0806 12:26:47.216895 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:47.216962 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:47.218287 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:47.218969 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:47.218979 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:47.219484 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:47.220193 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:47.220213 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.221052 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.221056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:47.222025 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:47.222038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.222577 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.222581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:47.222784 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.223410 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:47.223430 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:47.225060 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:47.226925 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:47.226933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:47.226955 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:47.227383 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:47.228413 (+ 1030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:47.228417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:47.228419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:47.228657 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:47.229215 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:47.229224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:47.229664 (+ 440us) client_negotiation.cc:769] Sending connection context
0806 12:26:47.229898 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 12:26:47.230316 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":169,"threads_started":1}
I20260806 12:26:47.233340 20846 heartbeater.cc:347] Connected to a master server at 127.28.210.126:34979
I20260806 12:26:47.233651 20846 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:47.234380 20846 heartbeater.cc:511] Master 127.28.210.126:34979 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:26:47.236969 20579 ts_manager.cc:194] Registered new tserver with Master: fc8ef18fff6b43e6b2422e21212148a3 (127.28.210.65:34221)
I20260806 12:26:47.238632 20579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:60035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:26:47.261452 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34979
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:47.383232 20856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:47.383533 20856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:47.383577 20856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:47.387773 20856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:47.387840 20856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:47.387909 20856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:26:47.393581 20856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:47.394956 20856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:47.396124 20856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:47.406131 20863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:47.406260 20856 server_base.cc:1034] running on GCE node
W20260806 12:26:47.406144 20864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:47.406201 20867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:47.406821 20856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:47.407469 20856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:47.408638 20856 hybrid_clock.cc:648] HybridClock initialized: now 1786019207408618 us; error 30 us; skew 500 ppm
Aug 06 12:26:47 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019207, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:47.411346 20856 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:26:47.412261 20856 webserver.cc:460] Webserver started at http://127.28.210.66:32987/ using document root <none> and password file <none>
I20260806 12:26:47.412830 20856 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:47.412899 20856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:47.413136 20856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:26:47.415460 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "75556d2cb84a4e24a80c2fc3015370bc"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "934df17d8661aba8757994bbbb2bc668"
server_key_iv: "ef8219566890a06c949d555498453c78"
server_key_version: "encryptionkey@0"
I20260806 12:26:47.416109 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "75556d2cb84a4e24a80c2fc3015370bc"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "934df17d8661aba8757994bbbb2bc668"
server_key_iv: "ef8219566890a06c949d555498453c78"
server_key_version: "encryptionkey@0"
I20260806 12:26:47.420543 20856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:26:47.423151 20876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:47.424576 20856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:26:47.424703 20856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "75556d2cb84a4e24a80c2fc3015370bc"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "934df17d8661aba8757994bbbb2bc668"
server_key_iv: "ef8219566890a06c949d555498453c78"
server_key_version: "encryptionkey@0"
I20260806 12:26:47.424821 20856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:47.439791 20856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:47.442827 20856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:47.443022 20856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:47.443625 20856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:47.444537 20856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:47.444597 20856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:47.444653 20856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:47.444679 20856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:47.453171 20856 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:34421
I20260806 12:26:47.453227 21042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:34421 every 8 connection(s)
I20260806 12:26:47.454104 20856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:26:47.459432 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 20856
I20260806 12:26:47.459564 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:26:47.459827 29513 external_mini_cluster.cc:1428] Setting key b967db57ac4b81825f53be919101ec42
Aug 06 12:26:47 dist-test-slave-lgr5 krb5kdc[20514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019207, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:47.468254 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.456024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:47805 (local address 127.28.210.126:34979)
0806 12:26:47.456183 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:47.456187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:47.457099 (+ 912us) server_negotiation.cc:407] Connection header received
0806 12:26:47.458472 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:47.458475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:47.458541 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:47.458622 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:47.459993 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.460672 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.461908 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.462139 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.462229 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.464812 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:47.464816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:47.464819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:47.464844 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:47.466100 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:47.466620 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:47.466624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:47.466626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:47.466709 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:47.467044 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:47.467047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:47.467049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:47.467246 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:47.467346 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:47.467891 (+ 545us) server_negotiation.cc:299] Negotiation successful
0806 12:26:47.468036 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 12:26:47.469165 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.456442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:47805)
0806 12:26:47.456929 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:47.456978 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:47.458241 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:47.458835 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:47.458843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:47.459242 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:47.459814 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:47.459831 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.460812 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.460815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:47.461789 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:47.461800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.462289 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.462293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:47.462454 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.463055 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:47.463072 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:47.464628 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:47.466219 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:47.466226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:47.466243 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:47.466521 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:47.466796 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:47.466799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:47.466802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:47.466950 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:47.467362 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:47.467370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:47.467648 (+ 278us) client_negotiation.cc:769] Sending connection context
0806 12:26:47.467832 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 12:26:47.468246 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":129,"threads_started":1}
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
I20260806 12:26:47.470372 21043 heartbeater.cc:347] Connected to a master server at 127.28.210.126:34979
I20260806 12:26:47.470633 21043 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:47.471179 21043 heartbeater.cc:511] Master 127.28.210.126:34979 requested a full tablet report, sending...
I20260806 12:26:47.472383 20579 ts_manager.cc:194] Registered new tserver with Master: 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
I20260806 12:26:47.473006 20579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:47805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:26:47.492202 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34979
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:26:47.606204 21054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:26:47.606472 21054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:26:47.606523 21054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:26:47.610610 21054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:26:47.610671 21054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:26:47.610729 21054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:26:47.616017 21054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:26:47.617146 21054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:26:47.618341 21054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:26:47.627990 21066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:47.627977 21061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:26:47.627981 21063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:26:47.628430 21054 server_base.cc:1034] running on GCE node
I20260806 12:26:47.628855 21054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:26:47.629575 21054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:26:47.630751 21054 hybrid_clock.cc:648] HybridClock initialized: now 1786019207630723 us; error 43 us; skew 500 ppm
Aug 06 12:26:47 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019207, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:26:47.633499 21054 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:26:47.634331 21054 webserver.cc:460] Webserver started at http://127.28.210.67:37213/ using document root <none> and password file <none>
I20260806 12:26:47.634856 21054 fs_manager.cc:359] Metadata directory not provided
I20260806 12:26:47.634910 21054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:26:47.635098 21054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:26:47.637148 21054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "2a29258d81234ab29370538ce0b88904"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "0fcad59ac18352cd2a60add840b1d254"
server_key_iv: "31dec7b0e796a1d2f972258060b3408d"
server_key_version: "encryptionkey@0"
I20260806 12:26:47.637673 21054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "2a29258d81234ab29370538ce0b88904"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "0fcad59ac18352cd2a60add840b1d254"
server_key_iv: "31dec7b0e796a1d2f972258060b3408d"
server_key_version: "encryptionkey@0"
I20260806 12:26:47.641407 21054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:26:47.643777 21078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:47.645097 21054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 12:26:47.645200 21054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "2a29258d81234ab29370538ce0b88904"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "0fcad59ac18352cd2a60add840b1d254"
server_key_iv: "31dec7b0e796a1d2f972258060b3408d"
server_key_version: "encryptionkey@0"
I20260806 12:26:47.645295 21054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:26:47.659991 21054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:26:47.663178 21054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:26:47.663362 21054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:26:47.663975 21054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:26:47.664934 21054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:26:47.664984 21054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:47.665025 21054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:26:47.665045 21054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:26:47.673936 21054 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:36601
I20260806 12:26:47.674014 21245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:36601 every 8 connection(s)
I20260806 12:26:47.674870 21054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:26:47.680020 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 21054
I20260806 12:26:47.680188 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:26:47.680554 29513 external_mini_cluster.cc:1428] Setting key 25e0ffb0eba978e7004a87f26a9bf87e
Aug 06 12:26:47 dist-test-slave-lgr5 krb5kdc[20514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019207, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:26:47.689612 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.676806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:57951 (local address 127.28.210.126:34979)
0806 12:26:47.676982 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:47.676987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:47.677907 (+ 920us) server_negotiation.cc:407] Connection header received
0806 12:26:47.679314 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:47.679338 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:47.679395 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:47.679481 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:47.681167 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.681767 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.683027 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.683251 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.683340 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.685969 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:47.685972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:47.685974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:47.686022 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:47.687388 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:47.687932 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:47.687935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:47.687936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:47.688026 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:47.688392 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:47.688396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:47.688398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:47.688597 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:47.688685 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:47.689280 (+ 595us) server_negotiation.cc:299] Negotiation successful
0806 12:26:47.689432 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 12:26:47.690514 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.677186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:57951)
0806 12:26:47.677721 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:47.677785 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:47.679043 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:47.679718 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:47.679728 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:47.680283 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:26:47.681001 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:47.681021 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.681911 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.681914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:47.682897 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:47.682909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.683409 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.683414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:47.683585 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.684190 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:26:47.684207 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:26:47.685783 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:26:47.687527 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:47.687533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:47.687553 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:47.687829 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:47.688137 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:26:47.688140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:26:47.688142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:26:47.688301 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:26:47.688700 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:26:47.688708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:26:47.689004 (+ 296us) client_negotiation.cc:769] Sending connection context
0806 12:26:47.689194 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 12:26:47.689590 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":133,"threads_started":1}
I20260806 12:26:47.691773 21246 heartbeater.cc:347] Connected to a master server at 127.28.210.126:34979
I20260806 12:26:47.692021 21246 heartbeater.cc:464] Registering TS with master...
I20260806 12:26:47.692680 21246 heartbeater.cc:511] Master 127.28.210.126:34979 requested a full tablet report, sending...
I20260806 12:26:47.693781 20578 ts_manager.cc:194] Registered new tserver with Master: 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601)
I20260806 12:26:47.694592 20578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:57951
I20260806 12:26:47.694669 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:26:47.704140 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.696678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.28.210.126:34979)
0806 12:26:47.696806 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:47.696811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:47.696985 (+ 174us) server_negotiation.cc:407] Connection header received
0806 12:26:47.697099 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:47.697103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:47.697146 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:47.697211 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:26:47.697920 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.698436 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.699270 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.699531 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.699624 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.701023 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:26:47.701027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:26:47.701029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:26:47.701056 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:26:47.702406 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:47.702818 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:47.702822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:47.702824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:47.702911 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:26:47.703164 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:26:47.703167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:26:47.703169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:26:47.703450 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:26:47.703577 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:47.703825 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 12:26:47.703948 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 12:26:47.707688 20578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:26:47.711357 20578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:26:47.725832 21265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.720048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:34221 (local address 127.0.0.1:48828)
0806 12:26:47.720928 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:47.720981 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:47.721108 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:47.721610 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:47.721616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:47.721640 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:47.721786 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:47.721792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.723044 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.723047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:47.723903 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:47.723915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.725230 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.725234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:47.725397 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.725443 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 12:26:47.725549 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 12:26:47.725678 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":23,"thread_start_us":106,"threads_started":1}
I20260806 12:26:47.726253 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.720228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:36601 (local address 127.0.0.1:45042)
0806 12:26:47.721640 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:47.721652 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:47.721793 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:47.722422 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:47.722425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:47.722442 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:47.722584 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:47.722592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.723929 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.723932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:47.724894 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:47.724904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.725844 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.725848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:47.725940 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.725964 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 12:26:47.726056 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 12:26:47.726124 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1355,"mutex_wait_us":33,"thread_start_us":100,"threads_started":1}
I20260806 12:26:47.726348 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.720924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.28.210.67:36601)
0806 12:26:47.721511 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:47.721519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:47.721730 (+ 211us) server_negotiation.cc:407] Connection header received
0806 12:26:47.721933 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:47.721937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:47.722131 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:47.722249 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:47.722703 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.723791 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.725018 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.725552 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.725655 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.725780 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:47.726112 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 12:26:47.726210 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":148,"threads_started":1}
I20260806 12:26:47.725875 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.720219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.28.210.65:34221)
0806 12:26:47.721120 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:47.721137 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:47.721163 (+ 26us) server_negotiation.cc:407] Connection header received
0806 12:26:47.721241 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:47.721246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:47.721442 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:47.721606 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:47.721910 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.722925 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.724031 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.724917 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.725021 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.725171 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:47.725613 (+ 442us) server_negotiation.cc:299] Negotiation successful
0806 12:26:47.725723 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":95,"threads_started":1}
I20260806 12:26:47.726994 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.720538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:34421 (local address 127.0.0.1:49146)
0806 12:26:47.721870 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:47.721882 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:47.721979 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:47.722338 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:47.722340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:47.722358 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:47.722482 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:47.722489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.724290 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.724293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:47.725401 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:47.725411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.726614 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.726617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:47.726701 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.726723 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 12:26:47.726800 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 12:26:47.726866 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"mutex_wait_us":24,"thread_start_us":112,"threads_started":1}
I20260806 12:26:47.727540 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:47.721457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.28.210.66:34421)
0806 12:26:47.721922 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:47.721933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:47.721955 (+ 22us) server_negotiation.cc:407] Connection header received
0806 12:26:47.722083 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:47.722086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:47.722219 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:47.722341 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:47.722589 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.723721 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.725536 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:47.726286 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:47.726407 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:47.726543 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:47.726852 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 12:26:47.726947 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":105,"threads_started":1}
I20260806 12:26:47.729352 21153 tablet_service.cc:1467] Processing CreateTablet for tablet 9b54743350864e3c9640ab92069a18c2 (DEFAULT_TABLE table=test-table [id=cf083303ec464a698d01e4493f62575f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:26:47.729350 20753 tablet_service.cc:1467] Processing CreateTablet for tablet 9b54743350864e3c9640ab92069a18c2 (DEFAULT_TABLE table=test-table [id=cf083303ec464a698d01e4493f62575f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:26:47.730327 20753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b54743350864e3c9640ab92069a18c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:47.730336 21153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b54743350864e3c9640ab92069a18c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:47.735119 20948 tablet_service.cc:1467] Processing CreateTablet for tablet 9b54743350864e3c9640ab92069a18c2 (DEFAULT_TABLE table=test-table [id=cf083303ec464a698d01e4493f62575f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:26:47.736372 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b54743350864e3c9640ab92069a18c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:26:47.736649 21272 tablet_bootstrap.cc:492] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904: Bootstrap starting.
I20260806 12:26:47.738122 21273 tablet_bootstrap.cc:492] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Bootstrap starting.
I20260806 12:26:47.739962 21272 tablet_bootstrap.cc:654] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:47.740306 21273 tablet_bootstrap.cc:654] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:47.740756 21272 log.cc:824] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:47.741120 21273 log.cc:824] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:47.742606 21272 tablet_bootstrap.cc:492] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904: No bootstrap required, opened a new log
I20260806 12:26:47.742810 21272 ts_tablet_manager.cc:1397] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 12:26:47.743755 21278 tablet_bootstrap.cc:492] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc: Bootstrap starting.
I20260806 12:26:47.746565 21278 tablet_bootstrap.cc:654] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc: Neither blocks nor log segments found. Creating new log.
I20260806 12:26:47.746691 21272 raft_consensus.cc:348] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } }
I20260806 12:26:47.747027 21272 raft_consensus.cc:374] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:47.747084 21272 raft_consensus.cc:729] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a29258d81234ab29370538ce0b88904, State: Initialized, Role: FOLLOWER
I20260806 12:26:47.747462 21278 log.cc:824] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc: Log is configured to *not* fsync() on all Append() calls
I20260806 12:26:47.747563 21272 consensus_queue.cc:260] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } }
I20260806 12:26:47.748286 21246 heartbeater.cc:503] Master 127.28.210.126:34979 was elected leader, sending a full tablet report...
I20260806 12:26:47.748617 21272 ts_tablet_manager.cc:1428] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 12:26:47.749950 21278 tablet_bootstrap.cc:492] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc: No bootstrap required, opened a new log
I20260806 12:26:47.750162 21278 ts_tablet_manager.cc:1397] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 12:26:47.750759 21273 tablet_bootstrap.cc:492] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: No bootstrap required, opened a new log
I20260806 12:26:47.751013 21273 ts_tablet_manager.cc:1397] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260806 12:26:47.753682 21278 raft_consensus.cc:348] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } }
I20260806 12:26:47.753935 21278 raft_consensus.cc:374] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:47.753994 21278 raft_consensus.cc:729] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75556d2cb84a4e24a80c2fc3015370bc, State: Initialized, Role: FOLLOWER
I20260806 12:26:47.754397 21278 consensus_queue.cc:260] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } }
I20260806 12:26:47.754897 21273 raft_consensus.cc:348] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } }
I20260806 12:26:47.755103 21043 heartbeater.cc:503] Master 127.28.210.126:34979 was elected leader, sending a full tablet report...
I20260806 12:26:47.755184 21273 raft_consensus.cc:374] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:26:47.755244 21273 raft_consensus.cc:729] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8ef18fff6b43e6b2422e21212148a3, State: Initialized, Role: FOLLOWER
I20260806 12:26:47.755426 21278 ts_tablet_manager.cc:1428] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 12:26:47.755671 21273 consensus_queue.cc:260] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } }
I20260806 12:26:47.756395 20846 heartbeater.cc:503] Master 127.28.210.126:34979 was elected leader, sending a full tablet report...
I20260806 12:26:47.757727 21273 ts_tablet_manager.cc:1428] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
W20260806 12:26:47.925940 21248 tablet.cc:2369] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:26:47.955294 21044 tablet.cc:2369] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:26:47.965157 20849 tablet.cc:2369] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:26:48.088624 21284 raft_consensus.cc:482] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:26:48.088851 21284 raft_consensus.cc:504] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } }
I20260806 12:26:48.090051 21284 leader_election.cc:288] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc8ef18fff6b43e6b2422e21212148a3 (127.28.210.65:34221), 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601)
I20260806 12:26:48.094414 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:48.090633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:41659 (local address 127.28.210.67:36601)
0806 12:26:48.090804 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:48.090809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:48.090920 (+ 111us) server_negotiation.cc:407] Connection header received
0806 12:26:48.091083 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:48.091086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:48.091142 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:48.091200 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:48.091510 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:48.092028 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:48.093362 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:48.093786 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:48.093904 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:48.093968 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:48.094162 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 12:26:48.094241 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 12:26:48.094405 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:48.090585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:59303 (local address 127.28.210.65:34221)
0806 12:26:48.090736 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:48.090740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:48.090999 (+ 259us) server_negotiation.cc:407] Connection header received
0806 12:26:48.091135 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:48.091138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:48.091186 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:48.091264 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:26:48.091545 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:48.092040 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:48.093362 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:48.093807 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:48.093925 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:48.093985 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:48.094181 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 12:26:48.094263 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 12:26:48.094424 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:48.090390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:36601 (local address 127.28.210.66:41659)
0806 12:26:48.090820 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:48.090832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:48.090959 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:48.091233 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:48.091236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:48.091253 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:48.091391 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:48.091404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:48.092182 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:48.092185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:48.093213 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:48.093227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:48.093918 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:48.093922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:48.094037 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:48.094064 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 12:26:48.094128 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 12:26:48.094251 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":75,"thread_start_us":99,"threads_started":1}
I20260806 12:26:48.094430 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:48.090390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:34221 (local address 127.28.210.66:59303)
0806 12:26:48.090886 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:48.090907 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 12:26:48.091019 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:26:48.091290 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:26:48.091293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:26:48.091310 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:26:48.091426 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:48.091433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:48.092180 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:48.092183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:48.093214 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:26:48.093227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:26:48.093938 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:26:48.093942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:26:48.094061 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:48.094079 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 12:26:48.094151 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 12:26:48.094272 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":137,"thread_start_us":99,"threads_started":1}
I20260806 12:26:48.095439 21178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b54743350864e3c9640ab92069a18c2" candidate_uuid: "75556d2cb84a4e24a80c2fc3015370bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a29258d81234ab29370538ce0b88904" is_pre_election: true
I20260806 12:26:48.095583 20780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b54743350864e3c9640ab92069a18c2" candidate_uuid: "75556d2cb84a4e24a80c2fc3015370bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8ef18fff6b43e6b2422e21212148a3" is_pre_election: true
I20260806 12:26:48.095806 21178 raft_consensus.cc:2454] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75556d2cb84a4e24a80c2fc3015370bc in term 0.
I20260806 12:26:48.095971 20780 raft_consensus.cc:2454] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75556d2cb84a4e24a80c2fc3015370bc in term 0.
I20260806 12:26:48.096422 20877 leader_election.cc:302] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a29258d81234ab29370538ce0b88904, 75556d2cb84a4e24a80c2fc3015370bc, fc8ef18fff6b43e6b2422e21212148a3; no voters:
I20260806 12:26:48.096791 21284 raft_consensus.cc:2781] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:26:48.096879 21284 raft_consensus.cc:482] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:26:48.096931 21284 raft_consensus.cc:3037] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:48.098019 21284 raft_consensus.cc:504] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } }
I20260806 12:26:48.098397 21284 leader_election.cc:288] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [CANDIDATE]: Term 1 election: Requested vote from peers fc8ef18fff6b43e6b2422e21212148a3 (127.28.210.65:34221), 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601)
I20260806 12:26:48.098888 21178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b54743350864e3c9640ab92069a18c2" candidate_uuid: "75556d2cb84a4e24a80c2fc3015370bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a29258d81234ab29370538ce0b88904"
I20260806 12:26:48.098874 20780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b54743350864e3c9640ab92069a18c2" candidate_uuid: "75556d2cb84a4e24a80c2fc3015370bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8ef18fff6b43e6b2422e21212148a3"
I20260806 12:26:48.099030 21178 raft_consensus.cc:3037] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:48.099030 20780 raft_consensus.cc:3037] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:26:48.100498 21178 raft_consensus.cc:2454] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75556d2cb84a4e24a80c2fc3015370bc in term 1.
I20260806 12:26:48.100498 20780 raft_consensus.cc:2454] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75556d2cb84a4e24a80c2fc3015370bc in term 1.
I20260806 12:26:48.100886 20877 leader_election.cc:302] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a29258d81234ab29370538ce0b88904, 75556d2cb84a4e24a80c2fc3015370bc; no voters:
I20260806 12:26:48.101094 21284 raft_consensus.cc:2781] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:26:48.101325 21284 raft_consensus.cc:686] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 1 LEADER]: Becoming Leader. State: Replica: 75556d2cb84a4e24a80c2fc3015370bc, State: Running, Role: LEADER
I20260806 12:26:48.101753 21284 consensus_queue.cc:237] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } }
I20260806 12:26:48.105115 20579 catalog_manager.cc:5495] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66). New cstate: current_term: 1 leader_uuid: "75556d2cb84a4e24a80c2fc3015370bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } health_report { overall_health: HEALTHY } } }
I20260806 12:26:48.180729 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:48.176948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49162 (local address 127.28.210.66:34421)
0806 12:26:48.177078 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 12:26:48.177082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:26:48.177097 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:26:48.177245 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:26:48.177247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:26:48.177346 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:26:48.177444 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:26:48.177740 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:48.178254 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:48.179144 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:26:48.179386 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:26:48.179467 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:26:48.179763 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:26:48.180278 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:26:48.180401 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 12:26:48.180546 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 12:26:48.180606 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 12:26:48.187670 21178 raft_consensus.cc:1261] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904 [term 1 FOLLOWER]: Refusing update from remote peer 75556d2cb84a4e24a80c2fc3015370bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:26:48.187670 20780 raft_consensus.cc:1261] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Refusing update from remote peer 75556d2cb84a4e24a80c2fc3015370bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:26:48.188567 21292 consensus_queue.cc:1035] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:26:48.188772 21284 consensus_queue.cc:1035] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:26:48.197463 21298 mvcc.cc:205] Tried to move back new op lower bound from 7315534676727455744 to 7315534676387504128. Current Snapshot: MvccSnapshot[applied={T|T < 7315534676727455744}]
I20260806 12:26:48.198802 21299 mvcc.cc:205] Tried to move back new op lower bound from 7315534676727455744 to 7315534676387504128. Current Snapshot: MvccSnapshot[applied={T|T < 7315534676727455744}]
I20260806 12:26:48.206807 20577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:26:48.206979 20577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:26:48.209467 20577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Sending DeleteTablet for 3 replicas of tablet 9b54743350864e3c9640ab92069a18c2
I20260806 12:26:48.210301 21153 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b54743350864e3c9640ab92069a18c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:26:48 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:45042
I20260806 12:26:48.210309 20753 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b54743350864e3c9640ab92069a18c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:26:48 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:48828
I20260806 12:26:48.210592 20948 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b54743350864e3c9640ab92069a18c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:26:48 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:49146
I20260806 12:26:48.210906 21307 tablet_replica.cc:330] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904: stopping tablet replica
I20260806 12:26:48.211030 21309 tablet_replica.cc:330] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc: stopping tablet replica
I20260806 12:26:48.211202 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 20662
I20260806 12:26:48.211243 21307 raft_consensus.cc:2229] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:26:48.211311 21309 raft_consensus.cc:2229] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:26:48.211583 21307 raft_consensus.cc:2258] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:26:48.211695 21309 raft_consensus.cc:2258] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:26:48.218434 21307 ts_tablet_manager.cc:1905] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:26:48.218492 21309 ts_tablet_manager.cc:1905] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:26:48.221997 21307 ts_tablet_manager.cc:1918] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:26:48.222110 21307 log.cc:1196] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/9b54743350864e3c9640ab92069a18c2
I20260806 12:26:48.222271 21309 ts_tablet_manager.cc:1918] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:26:48.222406 21307 ts_tablet_manager.cc:1939] T 9b54743350864e3c9640ab92069a18c2 P 2a29258d81234ab29370538ce0b88904: Deleting consensus metadata
I20260806 12:26:48.222398 21309 log.cc:1196] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/9b54743350864e3c9640ab92069a18c2
I20260806 12:26:48.222836 21309 ts_tablet_manager.cc:1939] T 9b54743350864e3c9640ab92069a18c2 P 75556d2cb84a4e24a80c2fc3015370bc: Deleting consensus metadata
I20260806 12:26:48.223865 20554 catalog_manager.cc:4841] TS 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601): tablet 9b54743350864e3c9640ab92069a18c2 (table test-table [id=cf083303ec464a698d01e4493f62575f]) successfully deleted
I20260806 12:26:48.231000 20557 catalog_manager.cc:4841] TS 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421): tablet 9b54743350864e3c9640ab92069a18c2 (table test-table [id=cf083303ec464a698d01e4493f62575f]) successfully deleted
W20260806 12:26:48.238515 20556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:34221 (error 108)
I20260806 12:26:48.238698 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 20856
I20260806 12:26:48.248530 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:26:48.248010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:34221 (local address 127.0.0.1:48838)
0806 12:26:48.248292 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 12:26:48.248390 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:34221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":94,"threads_started":1}
W20260806 12:26:48.248765 20556 catalog_manager.cc:4584] TS fc8ef18fff6b43e6b2422e21212148a3 (127.28.210.65:34221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b54743350864e3c9640ab92069a18c2: Network error: Client connection negotiation failed: client connection to 127.28.210.65:34221: connect: Connection refused (error 111)
I20260806 12:26:48.254680 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 21054
I20260806 12:26:48.267112 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 20529
I20260806 12:27:08.282876 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34979
--webserver_interface=127.28.210.126
--webserver_port=45061
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:34979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:08.406157 21313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:08.406545 21313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:08.406610 21313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:08.411041 21313 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 12:27:08.411123 21313 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 12:27:08.411145 21313 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 12:27:08.411166 21313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:27:08.411190 21313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:08.411214 21313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:08.411242 21313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:27:08.411273 21313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:27:08.417367 21313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:34979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=45061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:08.418630 21313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:08.419730 21313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:08.434888 21323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:08.434959 21313 server_base.cc:1034] running on GCE node
W20260806 12:27:08.434899 21322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:08.434978 21327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:08.435906 21313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:08.437317 21313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:08.438504 21313 hybrid_clock.cc:648] HybridClock initialized: now 1786019228438482 us; error 37 us; skew 500 ppm
Aug 06 12:27:08 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:08.441660 21313 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:27:08.442739 21313 webserver.cc:460] Webserver started at http://127.28.210.126:45061/ using document root <none> and password file <none>
I20260806 12:27:08.443506 21313 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:08.443610 21313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:08.448132 21313 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260806 12:27:08.450594 21335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:08.452046 21313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:27:08.452217 21313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "5cefe36342cc44e387733f3108482125"
format_stamp: "Formatted at 2026-08-06 12:26:46 on dist-test-slave-lgr5"
server_key: "535fc38864e16fca80c83a6d6a2a2aa1"
server_key_iv: "757c9d8208d0095e51afac08bcae7e30"
server_key_version: "encryptionkey@0"
I20260806 12:27:08.452937 21313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:08.476449 21313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:08.480468 21313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:08.480664 21313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:08.488842 21313 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:34979
I20260806 12:27:08.489148 21414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:34979 every 8 connection(s)
I20260806 12:27:08.490175 21313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:27:08.492168 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 21313
I20260806 12:27:08.492619 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:34221
--local_ip_for_outbound_sockets=127.28.210.65
--tserver_master_addrs=127.28.210.126:34979
--webserver_port=33697
--webserver_interface=127.28.210.65
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:27:08.495530 21416 sys_catalog.cc:263] Verifying existing consensus state
I20260806 12:27:08.497642 21416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Bootstrap starting.
I20260806 12:27:08.516049 21416 log.cc:824] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:08.525455 21416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 12:27:08.525928 21416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Bootstrap complete.
I20260806 12:27:08.530139 21416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } }
I20260806 12:27:08.530560 21416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cefe36342cc44e387733f3108482125, State: Initialized, Role: FOLLOWER
I20260806 12:27:08.531083 21416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } }
I20260806 12:27:08.531221 21416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:27:08.531301 21416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:27:08.531400 21416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 1 FOLLOWER]: Advancing to term 2
I20260806 12:27:08.533517 21416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } }
I20260806 12:27:08.533864 21416 leader_election.cc:302] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cefe36342cc44e387733f3108482125; no voters:
I20260806 12:27:08.534196 21416 leader_election.cc:288] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 12:27:08.534310 21425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 12:27:08.534524 21425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 2 LEADER]: Becoming Leader. State: Replica: 5cefe36342cc44e387733f3108482125, State: Running, Role: LEADER
I20260806 12:27:08.534828 21425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } }
I20260806 12:27:08.535568 21416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:27:08.536362 21428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cefe36342cc44e387733f3108482125. Latest consensus state: current_term: 2 leader_uuid: "5cefe36342cc44e387733f3108482125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } } }
I20260806 12:27:08.536402 21427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5cefe36342cc44e387733f3108482125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } } }
I20260806 12:27:08.536511 21428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:08.536543 21427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:08.536933 21435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:27:08.542315 21435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:27:08.543085 21435 catalog_manager.cc:1259] Loaded cluster ID: 577593c98d6445ba9e0a9ce7778398e1
I20260806 12:27:08.543249 21435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:27:08.545966 21435 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:27:08.546780 21435 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Loaded TSK: 0
I20260806 12:27:08.563328 21435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Generated new TSK 1
I20260806 12:27:08.563795 21435 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 12:27:08.620114 21423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:08.620402 21423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:08.620468 21423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:08.624532 21423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:08.624607 21423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:08.624688 21423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:27:08.630266 21423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:34221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=33697
--tserver_master_addrs=127.28.210.126:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:08.631318 21423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:08.632375 21423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:08.643471 21464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:08.643450 21460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:08.643453 21459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:08.643865 21423 server_base.cc:1034] running on GCE node
I20260806 12:27:08.644304 21423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:08.644955 21423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:08.646147 21423 hybrid_clock.cc:648] HybridClock initialized: now 1786019228646120 us; error 42 us; skew 500 ppm
Aug 06 12:27:08 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:08.648921 21423 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:27:08.649883 21423 webserver.cc:460] Webserver started at http://127.28.210.65:33697/ using document root <none> and password file <none>
I20260806 12:27:08.650455 21423 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:08.650533 21423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:08.654506 21423 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:27:08.656772 21474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:08.658170 21423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:08.658293 21423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "fc8ef18fff6b43e6b2422e21212148a3"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "5a24654cf2364cf6a88a39b77ab94679"
server_key_iv: "405768457ada9e32ac963f66507d5140"
server_key_version: "encryptionkey@0"
I20260806 12:27:08.658880 21423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:08.676833 21423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:08.680256 21423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:08.680457 21423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:08.681128 21423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:08.682300 21487 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 12:27:08.686836 21423 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 12:27:08.686929 21423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 12:27:08.687000 21423 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 12:27:08.688999 21423 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 12:27:08.689090 21423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260806 12:27:08.689400 21487 tablet_bootstrap.cc:492] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Bootstrap starting.
I20260806 12:27:08.700776 21423 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:34221
I20260806 12:27:08.700803 21638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:34221 every 8 connection(s)
I20260806 12:27:08.702214 21423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:27:08.711270 21487 log.cc:824] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:08.711306 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 21423
I20260806 12:27:08.711843 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:34421
--local_ip_for_outbound_sockets=127.28.210.66
--tserver_master_addrs=127.28.210.126:34979
--webserver_port=32987
--webserver_interface=127.28.210.66
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 12:27:08 dist-test-slave-lgr5 krb5kdc[20514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:27:08.719448 21487 tablet_bootstrap.cc:492] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 12:27:08.719898 21487 tablet_bootstrap.cc:492] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Bootstrap complete.
I20260806 12:27:08.721411 21487 ts_tablet_manager.cc:1397] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20260806 12:27:08.724516 21487 raft_consensus.cc:348] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } }
I20260806 12:27:08.725159 21487 raft_consensus.cc:729] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8ef18fff6b43e6b2422e21212148a3, State: Initialized, Role: FOLLOWER
I20260806 12:27:08.725459 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:08.704596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:34581)
0806 12:27:08.705203 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:08.705249 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:08.706095 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:08.707352 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:08.707360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:08.707832 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:08.708352 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:08.708374 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:08.709795 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:08.709798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:08.710733 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:08.710746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:08.711830 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:08.711835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:08.712060 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:08.716299 (+ 4239us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:08.716323 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:08.718442 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:08.721221 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:08.721229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:08.721244 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:08.721633 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:08.722148 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:08.722152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:08.722154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:08.722353 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:08.723161 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:08.723169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:08.723549 (+ 380us) client_negotiation.cc:769] Sending connection context
0806 12:27:08.723882 (+ 333us) client_negotiation.cc:240] Negotiation successful
0806 12:27:08.724339 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":197,"threads_started":1}
I20260806 12:27:08.725814 21487 consensus_queue.cc:260] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } }
I20260806 12:27:08.726220 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:08.704549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:34581 (local address 127.28.210.126:34979)
0806 12:27:08.705037 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:08.705060 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:08.705378 (+ 318us) server_negotiation.cc:407] Connection header received
0806 12:27:08.706569 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:08.706581 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:08.706909 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:08.707254 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:08.708560 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:08.709605 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:08.710881 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:08.711625 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:08.711807 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:08.718769 (+ 6962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:08.718778 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:08.718795 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:08.718837 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:08.721056 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:08.721799 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:08.721806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:08.721812 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:08.722031 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:08.722474 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:08.722477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:08.722479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:08.723019 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:08.723166 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:08.724062 (+ 896us) server_negotiation.cc:299] Negotiation successful
0806 12:27:08.724460 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":118,"threads_started":1}
I20260806 12:27:08.726948 21487 ts_tablet_manager.cc:1428] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 12:27:08.729058 21640 heartbeater.cc:347] Connected to a master server at 127.28.210.126:34979
I20260806 12:27:08.729312 21640 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:08.729887 21640 heartbeater.cc:511] Master 127.28.210.126:34979 requested a full tablet report, sending...
I20260806 12:27:08.732446 21363 ts_manager.cc:194] Registered new tserver with Master: fc8ef18fff6b43e6b2422e21212148a3 (127.28.210.65:34221)
I20260806 12:27:08.735869 21363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:34581
I20260806 12:27:08.738591 21640 heartbeater.cc:503] Master 127.28.210.126:34979 was elected leader, sending a full tablet report...
Aug 06 12:27:08 dist-test-slave-lgr5 krb5kdc[20514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.65@KRBTEST.COM
I20260806 12:27:08.746649 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:08.735413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.28.210.65:34221)
0806 12:27:08.735883 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:08.735895 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:08.736422 (+ 527us) server_negotiation.cc:407] Connection header received
0806 12:27:08.736628 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:08.736633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:08.736844 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:08.737076 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:08.737756 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:08.738840 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:08.740045 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:08.740490 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:08.740576 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:08.743124 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:08.743135 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:08.743147 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:08.743171 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:08.744713 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:08.745205 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:08.745210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:08.745211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:08.745307 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:08.745573 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:08.745577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:08.745578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:08.745844 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:08.745966 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:08.746326 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 12:27:08.746494 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":159,"threads_started":1}
I20260806 12:27:08.746650 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:08.735322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:34221 (local address 127.0.0.1:42852)
0806 12:27:08.736267 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:08.736316 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:08.736470 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:08.737035 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:08.737041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:08.737476 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:08.737617 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:08.737627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:08.738986 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:08.738989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:08.739929 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:08.739941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:08.740607 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:08.740611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:08.740723 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:08.741307 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:08.741322 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:08.742955 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:08.744837 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:08.744847 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:08.744853 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:08.745089 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:08.745398 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:08.745401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:08.745403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:08.745487 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:08.745960 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:08.745966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:08.746092 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 12:27:08.746217 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 12:27:08.746439 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":108,"threads_started":1}
I20260806 12:27:08.747962 21544 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b54743350864e3c9640ab92069a18c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:26:48 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:42852
I20260806 12:27:08.748996 21652 tablet_replica.cc:330] stopping tablet replica
I20260806 12:27:08.749358 21652 raft_consensus.cc:2229] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:27:08.749639 21652 raft_consensus.cc:2258] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:08.750846 21652 ts_tablet_manager.cc:1905] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:08.756296 21652 ts_tablet_manager.cc:1918] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:08.756443 21652 log.cc:1196] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal/wals/9b54743350864e3c9640ab92069a18c2
I20260806 12:27:08.756734 21652 ts_tablet_manager.cc:1939] T 9b54743350864e3c9640ab92069a18c2 P fc8ef18fff6b43e6b2422e21212148a3: Deleting consensus metadata
I20260806 12:27:08.759035 21337 catalog_manager.cc:4841] TS fc8ef18fff6b43e6b2422e21212148a3 (127.28.210.65:34221): tablet 9b54743350864e3c9640ab92069a18c2 (table test-table [id=cf083303ec464a698d01e4493f62575f]) successfully deleted
W20260806 12:27:08.840590 21646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:08.840867 21646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:08.840915 21646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:08.845227 21646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:08.845319 21646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:08.845386 21646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:27:08.851223 21646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:34421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=32987
--tserver_master_addrs=127.28.210.126:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:08.852451 21646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:08.853525 21646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:08.863636 21659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:08.863655 21662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:08.863644 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:08.863962 21646 server_base.cc:1034] running on GCE node
I20260806 12:27:08.864444 21646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:08.865159 21646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:08.866348 21646 hybrid_clock.cc:648] HybridClock initialized: now 1786019228866325 us; error 32 us; skew 500 ppm
Aug 06 12:27:08 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:08.869264 21646 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:27:08.870208 21646 webserver.cc:460] Webserver started at http://127.28.210.66:32987/ using document root <none> and password file <none>
I20260806 12:27:08.870790 21646 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:08.870877 21646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:08.874984 21646 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260806 12:27:08.877285 21678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:08.878757 21646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:08.878903 21646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "75556d2cb84a4e24a80c2fc3015370bc"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "934df17d8661aba8757994bbbb2bc668"
server_key_iv: "ef8219566890a06c949d555498453c78"
server_key_version: "encryptionkey@0"
I20260806 12:27:08.879509 21646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:08.902290 21646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:08.905643 21646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:08.905850 21646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:08.906487 21646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:08.907416 21646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:08.907466 21646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:08.907521 21646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:08.907546 21646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:08.916561 21646 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:34421
I20260806 12:27:08.916592 21847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:34421 every 8 connection(s)
I20260806 12:27:08.917605 21646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:27:08.920364 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 21646
I20260806 12:27:08.920848 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:36601
--local_ip_for_outbound_sockets=127.28.210.67
--tserver_master_addrs=127.28.210.126:34979
--webserver_port=37213
--webserver_interface=127.28.210.67
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 12:27:08 dist-test-slave-lgr5 krb5kdc[20514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019228, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:27:08.933267 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:08.919501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:46595 (local address 127.28.210.126:34979)
0806 12:27:08.919683 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:08.919687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:08.920505 (+ 818us) server_negotiation.cc:407] Connection header received
0806 12:27:08.921952 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:08.921955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:08.922020 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:08.922109 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:08.923665 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:08.924254 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:08.925896 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:08.926187 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:08.926304 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:08.929382 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:08.929386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:08.929388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:08.929419 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:08.930908 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:08.931505 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:08.931509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:08.931510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:08.931604 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:08.931986 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:08.931990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:08.931992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:08.932239 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:08.932338 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:08.932913 (+ 575us) server_negotiation.cc:299] Negotiation successful
0806 12:27:08.933073 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 12:27:08.934146 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:08.919841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:46595)
0806 12:27:08.920353 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:08.920397 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:08.921721 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:08.922273 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:08.922279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:08.922762 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:08.923470 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:08.923490 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:08.924394 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:08.924397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:08.925730 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:08.925742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:08.926369 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:08.926374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:08.926588 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:08.927253 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:08.927276 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:08.929172 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:08.931050 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:08.931057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:08.931076 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:08.931392 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:08.931699 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:08.931703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:08.931705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:08.931886 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:08.932365 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:08.932373 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:08.932654 (+ 281us) client_negotiation.cc:769] Sending connection context
0806 12:27:08.932853 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 12:27:08.933220 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":123,"threads_started":1}
I20260806 12:27:08.935559 21849 heartbeater.cc:347] Connected to a master server at 127.28.210.126:34979
I20260806 12:27:08.935828 21849 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:08.936503 21849 heartbeater.cc:511] Master 127.28.210.126:34979 requested a full tablet report, sending...
I20260806 12:27:08.937899 21363 ts_manager.cc:194] Registered new tserver with Master: 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421)
I20260806 12:27:08.938704 21363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:46595
W20260806 12:27:09.042716 21856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:09.043016 21856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:09.043061 21856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:09.047101 21856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:09.047170 21856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:09.047230 21856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:27:09.052820 21856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:36601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=37213
--tserver_master_addrs=127.28.210.126:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:09.054073 21856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:09.055203 21856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:09.065941 21864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:09.065949 21865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:09.066071 21856 server_base.cc:1034] running on GCE node
W20260806 12:27:09.065958 21869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:09.066771 21856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:09.067495 21856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:09.068696 21856 hybrid_clock.cc:648] HybridClock initialized: now 1786019229068655 us; error 49 us; skew 500 ppm
Aug 06 12:27:09 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019229, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:09.071933 21856 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:27:09.073014 21856 webserver.cc:460] Webserver started at http://127.28.210.67:37213/ using document root <none> and password file <none>
I20260806 12:27:09.073623 21856 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:09.073710 21856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:09.078001 21856 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:27:09.080413 21882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:09.081925 21856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 12:27:09.082080 21856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "2a29258d81234ab29370538ce0b88904"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "0fcad59ac18352cd2a60add840b1d254"
server_key_iv: "31dec7b0e796a1d2f972258060b3408d"
server_key_version: "encryptionkey@0"
I20260806 12:27:09.082788 21856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:09.123626 21856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:09.127076 21856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:09.127293 21856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:09.127960 21856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:09.128909 21856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:09.128961 21856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:09.129019 21856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:09.129046 21856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:09.138790 21856 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:36601
I20260806 12:27:09.138844 22050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:36601 every 8 connection(s)
I20260806 12:27:09.140205 21856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:27:09.149957 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 21856
Aug 06 12:27:09 dist-test-slave-lgr5 krb5kdc[20514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019229, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:27:09.157164 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.142519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:52227 (local address 127.28.210.126:34979)
0806 12:27:09.142716 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:09.142721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:09.143817 (+ 1096us) server_negotiation.cc:407] Connection header received
0806 12:27:09.145264 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:09.145267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:09.145338 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:09.145427 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:09.146742 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.147402 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.148715 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.148989 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.149098 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.152468 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:09.152472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:09.152474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:09.152505 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:09.154239 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:09.155050 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:09.155054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:09.155056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:09.155173 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:09.155686 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:09.155690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:09.155692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:09.155982 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:09.156132 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:09.156818 (+ 686us) server_negotiation.cc:299] Negotiation successful
0806 12:27:09.156965 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 12:27:09.158494 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.143026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:52227)
0806 12:27:09.143631 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:09.143684 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:09.145006 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:09.145606 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:09.145613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:09.146035 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:09.146569 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:09.146586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.147575 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.147578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:09.148565 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:09.148577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.149168 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.149171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:09.149349 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.150068 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:09.150087 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:09.152275 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:09.154387 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:09.154396 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:09.154420 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:09.154891 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:09.155311 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:09.155315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:09.155318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:09.155564 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:09.156129 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:09.156137 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:09.156487 (+ 350us) client_negotiation.cc:769] Sending connection context
0806 12:27:09.156706 (+ 219us) client_negotiation.cc:240] Negotiation successful
0806 12:27:09.157141 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":163,"threads_started":1}
I20260806 12:27:09.159894 22051 heartbeater.cc:347] Connected to a master server at 127.28.210.126:34979
I20260806 12:27:09.160056 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.150762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.28.210.126:34979)
0806 12:27:09.151148 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:09.151152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:09.151167 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:27:09.151301 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:09.151304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:09.151359 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:09.151449 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:09.152424 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.153024 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.154287 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.154666 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.154790 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.156599 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:09.156603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:09.156606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:09.156628 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:09.158287 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:09.158758 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:09.158762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:09.158764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:09.158865 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:09.159168 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:09.159172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:09.159173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:09.159388 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:09.159590 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:09.159749 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 12:27:09.159898 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":127,"threads_started":1}
I20260806 12:27:09.160218 22051 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:09.160876 22051 heartbeater.cc:511] Master 127.28.210.126:34979 requested a full tablet report, sending...
I20260806 12:27:09.162143 21363 ts_manager.cc:194] Registered new tserver with Master: 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601)
I20260806 12:27:09.162715 21363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:52227
I20260806 12:27:09.173758 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:27:09.177670 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.174371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.28.210.126:34979)
0806 12:27:09.174499 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:09.174502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:09.174687 (+ 185us) server_negotiation.cc:407] Connection header received
0806 12:27:09.174835 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:09.174837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:09.174955 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:09.175051 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:27:09.175254 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.175777 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.176661 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.176878 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.176960 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.177198 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:27:09.177344 (+ 146us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 12:27:09.177557 (+ 213us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45740: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 12:27:09.177734 22057 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45740: token expired
W20260806 12:27:09.178148 21251 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.210.126:34979)
I20260806 12:27:09.185604 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.178532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.28.210.126:34979)
0806 12:27:09.178652 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:09.178656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:09.178668 (+ 12us) server_negotiation.cc:407] Connection header received
0806 12:27:09.178763 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:09.178766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:09.178802 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:09.178858 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:09.179451 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.179936 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.180964 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.181174 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.181258 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.182486 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:09.182490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:09.182493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:09.182517 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:09.183880 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:09.184369 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:09.184373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:09.184375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:09.184455 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:09.184701 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:09.184704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:09.184706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:09.184966 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:09.185083 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:09.185328 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 12:27:09.185452 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 12:27:09.188299 21363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:27:09.191586 21363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:27:09.199965 21544 tablet_service.cc:1467] Processing CreateTablet for tablet 18b26e0d9f754f8eb2b882e118a8bba0 (DEFAULT_TABLE table=test-table [id=64c22f15b829469c956b675bd8c612db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:09.200681 21544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b26e0d9f754f8eb2b882e118a8bba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:09.203588 21487 tablet_bootstrap.cc:492] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3: Bootstrap starting.
I20260806 12:27:09.204488 21487 tablet_bootstrap.cc:654] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:09.204892 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.198855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:34421 (local address 127.0.0.1:43602)
0806 12:27:09.199052 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:09.199068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:09.199154 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:09.200285 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:09.200288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:09.200307 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:09.200433 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:09.200440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.202046 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.202049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:09.203194 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:09.203206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.204515 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.204519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:09.204593 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.204611 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 12:27:09.204676 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 12:27:09.204755 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260806 12:27:09.204947 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.199335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43602 (local address 127.28.210.66:34421)
0806 12:27:09.199706 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:09.199719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:09.199745 (+ 26us) server_negotiation.cc:407] Connection header received
0806 12:27:09.199833 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:09.199839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:09.200109 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:09.200594 (+ 485us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:09.200765 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.201920 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.203325 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.204179 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.204285 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.204437 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:09.204726 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 12:27:09.204821 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":142,"threads_started":1}
I20260806 12:27:09.205096 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.199758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.28.210.67:36601)
0806 12:27:09.200196 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:09.200207 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:09.200230 (+ 23us) server_negotiation.cc:407] Connection header received
0806 12:27:09.200301 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:09.200306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:09.200475 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:09.200607 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:09.201034 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.202156 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.203167 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.204174 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.204285 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.204437 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:09.204854 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 12:27:09.204963 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":176,"threads_started":1}
I20260806 12:27:09.204772 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.199312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:36601 (local address 127.0.0.1:58598)
0806 12:27:09.199797 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:09.199810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:09.199900 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:09.200619 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:09.200622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:09.200641 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:09.200737 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:09.200744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.202259 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.202262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:09.203045 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:09.203053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.204293 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.204297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:09.204387 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.204451 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 12:27:09.204533 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 12:27:09.204614 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":2,"thread_start_us":116,"threads_started":1}
I20260806 12:27:09.206496 21487 tablet_bootstrap.cc:492] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3: No bootstrap required, opened a new log
I20260806 12:27:09.206589 21487 ts_tablet_manager.cc:1397] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 12:27:09.207010 21487 raft_consensus.cc:348] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:09.207113 21487 raft_consensus.cc:374] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:09.207149 21487 raft_consensus.cc:729] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8ef18fff6b43e6b2422e21212148a3, State: Initialized, Role: FOLLOWER
I20260806 12:27:09.207273 21487 consensus_queue.cc:260] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:09.207612 21487 ts_tablet_manager.cc:1428] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 12:27:09.208302 21952 tablet_service.cc:1467] Processing CreateTablet for tablet 18b26e0d9f754f8eb2b882e118a8bba0 (DEFAULT_TABLE table=test-table [id=64c22f15b829469c956b675bd8c612db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:09.208302 21755 tablet_service.cc:1467] Processing CreateTablet for tablet 18b26e0d9f754f8eb2b882e118a8bba0 (DEFAULT_TABLE table=test-table [id=64c22f15b829469c956b675bd8c612db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:09.209460 21755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b26e0d9f754f8eb2b882e118a8bba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:09.209478 21952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b26e0d9f754f8eb2b882e118a8bba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:09.218426 22072 tablet_bootstrap.cc:492] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904: Bootstrap starting.
I20260806 12:27:09.218451 22073 tablet_bootstrap.cc:492] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc: Bootstrap starting.
I20260806 12:27:09.221442 22073 tablet_bootstrap.cc:654] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:09.221442 22072 tablet_bootstrap.cc:654] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:09.222388 22072 log.cc:824] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:09.222388 22073 log.cc:824] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:09.224388 22072 tablet_bootstrap.cc:492] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904: No bootstrap required, opened a new log
I20260806 12:27:09.224592 22072 ts_tablet_manager.cc:1397] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 12:27:09.225131 22073 tablet_bootstrap.cc:492] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc: No bootstrap required, opened a new log
I20260806 12:27:09.225303 22073 ts_tablet_manager.cc:1397] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 12:27:09.227910 22072 raft_consensus.cc:348] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:09.228240 22072 raft_consensus.cc:374] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:09.228302 22072 raft_consensus.cc:729] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a29258d81234ab29370538ce0b88904, State: Initialized, Role: FOLLOWER
I20260806 12:27:09.228485 22073 raft_consensus.cc:348] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:09.228727 22073 raft_consensus.cc:374] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:09.228785 22073 raft_consensus.cc:729] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75556d2cb84a4e24a80c2fc3015370bc, State: Initialized, Role: FOLLOWER
I20260806 12:27:09.228790 22072 consensus_queue.cc:260] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:09.229202 22073 consensus_queue.cc:260] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:09.229593 22051 heartbeater.cc:503] Master 127.28.210.126:34979 was elected leader, sending a full tablet report...
I20260806 12:27:09.229835 21849 heartbeater.cc:503] Master 127.28.210.126:34979 was elected leader, sending a full tablet report...
I20260806 12:27:09.230020 22072 ts_tablet_manager.cc:1428] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 12:27:09.230202 22073 ts_tablet_manager.cc:1428] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 12:27:09.351480 22079 raft_consensus.cc:482] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:27:09.351718 22079 raft_consensus.cc:504] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:09.353057 22079 leader_election.cc:288] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421), fc8ef18fff6b43e6b2422e21212148a3 (127.28.210.65:34221)
I20260806 12:27:09.356715 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.353336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:34421 (local address 127.28.210.67:36621)
0806 12:27:09.353456 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:09.353472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:09.353572 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:09.353834 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:09.353837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:09.353860 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:09.353981 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:09.353987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.354711 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.354715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:09.355515 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:09.355525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.356295 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.356299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:09.356412 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.356431 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 12:27:09.356499 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 12:27:09.356574 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 12:27:09.356859 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.353451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:36621 (local address 127.28.210.66:34421)
0806 12:27:09.353574 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:09.353578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:09.353594 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:27:09.353649 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:09.353651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:09.353695 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:09.353783 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:09.354103 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.354605 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.355651 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.356176 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.356295 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.356355 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:09.356670 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 12:27:09.356747 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 12:27:09.357503 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.353593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:37201 (local address 127.28.210.65:34221)
0806 12:27:09.353883 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:09.353888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:09.354064 (+ 176us) server_negotiation.cc:407] Connection header received
0806 12:27:09.354190 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:09.354192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:09.354247 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:09.354334 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:09.354596 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.355108 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.356321 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.356953 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.357063 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.357228 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:09.357300 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 12:27:09.357371 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
I20260806 12:27:09.357523 22082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.353336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:34221 (local address 127.28.210.67:37201)
0806 12:27:09.353978 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:09.353988 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:09.354084 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:09.354355 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:09.354358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:09.354375 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:09.354495 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:09.354500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.355213 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.355216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:09.356182 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:09.356193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.357107 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.357111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:09.357200 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.357222 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 12:27:09.357287 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 12:27:09.357354 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":297,"thread_start_us":90,"threads_started":1}
I20260806 12:27:09.357506 21779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b26e0d9f754f8eb2b882e118a8bba0" candidate_uuid: "2a29258d81234ab29370538ce0b88904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75556d2cb84a4e24a80c2fc3015370bc" is_pre_election: true
I20260806 12:27:09.357874 21779 raft_consensus.cc:2454] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a29258d81234ab29370538ce0b88904 in term 0.
I20260806 12:27:09.358011 21576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b26e0d9f754f8eb2b882e118a8bba0" candidate_uuid: "2a29258d81234ab29370538ce0b88904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8ef18fff6b43e6b2422e21212148a3" is_pre_election: true
I20260806 12:27:09.358278 21576 raft_consensus.cc:2454] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a29258d81234ab29370538ce0b88904 in term 0.
I20260806 12:27:09.358369 21887 leader_election.cc:302] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a29258d81234ab29370538ce0b88904, 75556d2cb84a4e24a80c2fc3015370bc; no voters:
I20260806 12:27:09.358671 22079 raft_consensus.cc:2781] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:27:09.358752 22079 raft_consensus.cc:482] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:27:09.358783 22079 raft_consensus.cc:3037] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:09.359894 22079 raft_consensus.cc:504] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:09.360280 22079 leader_election.cc:288] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [CANDIDATE]: Term 1 election: Requested vote from peers 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421), fc8ef18fff6b43e6b2422e21212148a3 (127.28.210.65:34221)
I20260806 12:27:09.360661 21576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b26e0d9f754f8eb2b882e118a8bba0" candidate_uuid: "2a29258d81234ab29370538ce0b88904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8ef18fff6b43e6b2422e21212148a3"
I20260806 12:27:09.360806 21576 raft_consensus.cc:3037] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:09.360934 21779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b26e0d9f754f8eb2b882e118a8bba0" candidate_uuid: "2a29258d81234ab29370538ce0b88904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75556d2cb84a4e24a80c2fc3015370bc"
I20260806 12:27:09.361071 21779 raft_consensus.cc:3037] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:09.361963 21576 raft_consensus.cc:2454] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a29258d81234ab29370538ce0b88904 in term 1.
I20260806 12:27:09.362299 21779 raft_consensus.cc:2454] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a29258d81234ab29370538ce0b88904 in term 1.
I20260806 12:27:09.362304 21886 leader_election.cc:302] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a29258d81234ab29370538ce0b88904, fc8ef18fff6b43e6b2422e21212148a3; no voters:
I20260806 12:27:09.362514 22079 raft_consensus.cc:2781] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:09.362767 22079 raft_consensus.cc:686] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 1 LEADER]: Becoming Leader. State: Replica: 2a29258d81234ab29370538ce0b88904, State: Running, Role: LEADER
I20260806 12:27:09.363193 22079 consensus_queue.cc:237] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:09.366707 21360 catalog_manager.cc:5495] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a29258d81234ab29370538ce0b88904 (127.28.210.67). New cstate: current_term: 1 leader_uuid: "2a29258d81234ab29370538ce0b88904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } health_report { overall_health: UNKNOWN } } }
W20260806 12:27:09.391400 22052 tablet.cc:2369] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:27:09.402107 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.398741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.28.210.67:36601)
0806 12:27:09.398904 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:09.398908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:09.398922 (+ 14us) server_negotiation.cc:407] Connection header received
0806 12:27:09.399015 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:09.399017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:09.399073 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:09.399167 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:27:09.399369 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.399872 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.400695 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:09.400902 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:09.400986 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:09.401286 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:27:09.401742 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:27:09.401850 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:09.401897 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 12:27:09.401953 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 12:27:09.408177 21779 raft_consensus.cc:1261] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc [term 1 FOLLOWER]: Refusing update from remote peer 2a29258d81234ab29370538ce0b88904: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:09.408178 21576 raft_consensus.cc:1261] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Refusing update from remote peer 2a29258d81234ab29370538ce0b88904: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:09.408886 22079 consensus_queue.cc:1035] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:09.409085 22083 consensus_queue.cc:1035] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:09.416004 22089 mvcc.cc:205] Tried to move back new op lower bound from 7315534763648446464 to 7315534763474604032. Current Snapshot: MvccSnapshot[applied={T|T < 7315534763648446464}]
I20260806 12:27:09.416780 22088 mvcc.cc:205] Tried to move back new op lower bound from 7315534763648446464 to 7315534763474604032. Current Snapshot: MvccSnapshot[applied={T|T < 7315534763648446464}]
I20260806 12:27:09.426352 21360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:09.426568 21360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:09.429311 21360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Sending DeleteTablet for 3 replicas of tablet 18b26e0d9f754f8eb2b882e118a8bba0
I20260806 12:27:09.430198 21544 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b26e0d9f754f8eb2b882e118a8bba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:09 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:42852
I20260806 12:27:09.430217 21952 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b26e0d9f754f8eb2b882e118a8bba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:09 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:58598
I20260806 12:27:09.430222 21755 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b26e0d9f754f8eb2b882e118a8bba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:09 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:43602
I20260806 12:27:09.430816 22104 tablet_replica.cc:330] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc: stopping tablet replica
I20260806 12:27:09.431008 22103 tablet_replica.cc:330] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3: stopping tablet replica
I20260806 12:27:09.431126 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 21423
I20260806 12:27:09.431218 22104 raft_consensus.cc:2229] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:27:09.431175 22103 raft_consensus.cc:2229] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:27:09.431625 22104 raft_consensus.cc:2258] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:09.433924 22105 tablet_replica.cc:330] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904: stopping tablet replica
I20260806 12:27:09.434489 22105 raft_consensus.cc:2229] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:27:09.435189 22105 raft_consensus.cc:2258] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:09.437827 22104 ts_tablet_manager.cc:1905] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:09.439766 22105 ts_tablet_manager.cc:1905] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:09.440924 22104 ts_tablet_manager.cc:1918] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:09.441078 22104 log.cc:1196] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/18b26e0d9f754f8eb2b882e118a8bba0
I20260806 12:27:09.441506 22104 ts_tablet_manager.cc:1939] T 18b26e0d9f754f8eb2b882e118a8bba0 P 75556d2cb84a4e24a80c2fc3015370bc: Deleting consensus metadata
I20260806 12:27:09.442659 22105 ts_tablet_manager.cc:1918] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:09.442782 22105 log.cc:1196] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/18b26e0d9f754f8eb2b882e118a8bba0
I20260806 12:27:09.443135 22105 ts_tablet_manager.cc:1939] T 18b26e0d9f754f8eb2b882e118a8bba0 P 2a29258d81234ab29370538ce0b88904: Deleting consensus metadata
I20260806 12:27:09.443208 21339 catalog_manager.cc:4841] TS 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421): tablet 18b26e0d9f754f8eb2b882e118a8bba0 (table test-table [id=64c22f15b829469c956b675bd8c612db]) successfully deleted
I20260806 12:27:09.444566 21336 catalog_manager.cc:4841] TS 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601): tablet 18b26e0d9f754f8eb2b882e118a8bba0 (table test-table [id=64c22f15b829469c956b675bd8c612db]) successfully deleted
W20260806 12:27:09.448741 21337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:34221 (error 108)
I20260806 12:27:09.449311 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 21646
I20260806 12:27:09.468786 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:09.468319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:34221 (local address 127.0.0.1:42860)
0806 12:27:09.468580 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:09.468679 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:34221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260806 12:27:09.469226 21337 catalog_manager.cc:4584] TS fc8ef18fff6b43e6b2422e21212148a3 (127.28.210.65:34221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18b26e0d9f754f8eb2b882e118a8bba0: Network error: Client connection negotiation failed: client connection to 127.28.210.65:34221: connect: Connection refused (error 111)
I20260806 12:27:09.476444 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 21856
I20260806 12:27:09.488680 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 21313
I20260806 12:27:29.501034 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34979
--webserver_interface=127.28.210.126
--webserver_port=45061
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:34979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:29.637374 22108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:29.637725 22108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:29.637770 22108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:29.642524 22108 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 12:27:29.642630 22108 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 12:27:29.642648 22108 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 12:27:29.642663 22108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:27:29.642680 22108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:29.642699 22108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:29.642714 22108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:27:29.642731 22108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:27:29.649437 22108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:34979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=45061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:29.650923 22108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:29.652295 22108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:27:29.662364 22108 server_base.cc:1034] running on GCE node
W20260806 12:27:29.662384 22123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:29.662411 22124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:29.662441 22128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:29.663251 22108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:29.664642 22108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:29.666157 22108 hybrid_clock.cc:648] HybridClock initialized: now 1786019249666124 us; error 38 us; skew 500 ppm
Aug 06 12:27:29 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019249, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:29.669476 22108 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:27:29.670679 22108 webserver.cc:460] Webserver started at http://127.28.210.126:45061/ using document root <none> and password file <none>
I20260806 12:27:29.671319 22108 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:29.671394 22108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:29.676484 22108 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:29.679289 22138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:29.681133 22108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:29.681295 22108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "5cefe36342cc44e387733f3108482125"
format_stamp: "Formatted at 2026-08-06 12:26:46 on dist-test-slave-lgr5"
server_key: "535fc38864e16fca80c83a6d6a2a2aa1"
server_key_iv: "757c9d8208d0095e51afac08bcae7e30"
server_key_version: "encryptionkey@0"
I20260806 12:27:29.681963 22108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:29.693819 22108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:29.697395 22108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:29.697594 22108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:29.705873 22108 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:34979
I20260806 12:27:29.706223 22221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:34979 every 8 connection(s)
I20260806 12:27:29.707124 22108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:27:29.710757 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 22108
I20260806 12:27:29.711354 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:34221
--local_ip_for_outbound_sockets=127.28.210.65
--tserver_master_addrs=127.28.210.126:34979
--webserver_port=33697
--webserver_interface=127.28.210.65
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:27:29.712622 22222 sys_catalog.cc:263] Verifying existing consensus state
I20260806 12:27:29.714802 22222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Bootstrap starting.
I20260806 12:27:29.735394 22222 log.cc:824] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:29.750967 22222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 12:27:29.751473 22222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: Bootstrap complete.
I20260806 12:27:29.756723 22222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } }
I20260806 12:27:29.757287 22222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cefe36342cc44e387733f3108482125, State: Initialized, Role: FOLLOWER
I20260806 12:27:29.757838 22222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } }
I20260806 12:27:29.757969 22222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:27:29.758038 22222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:27:29.758139 22222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 2 FOLLOWER]: Advancing to term 3
I20260806 12:27:29.760429 22222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } }
I20260806 12:27:29.760797 22222 leader_election.cc:302] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cefe36342cc44e387733f3108482125; no voters:
I20260806 12:27:29.761152 22222 leader_election.cc:288] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 12:27:29.761269 22229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 12:27:29.761511 22229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [term 3 LEADER]: Becoming Leader. State: Replica: 5cefe36342cc44e387733f3108482125, State: Running, Role: LEADER
I20260806 12:27:29.761912 22229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } }
I20260806 12:27:29.762619 22222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:27:29.763676 22232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cefe36342cc44e387733f3108482125. Latest consensus state: current_term: 3 leader_uuid: "5cefe36342cc44e387733f3108482125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } } }
I20260806 12:27:29.763816 22232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:29.764106 22231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5cefe36342cc44e387733f3108482125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cefe36342cc44e387733f3108482125" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34979 } } }
I20260806 12:27:29.764190 22231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125 [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:29.768862 22253 catalog_manager.cc:1259] Loaded cluster ID: 577593c98d6445ba9e0a9ce7778398e1
I20260806 12:27:29.768946 22253 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: loading cluster ID for follower catalog manager: success
I20260806 12:27:29.771799 22253 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5cefe36342cc44e387733f3108482125: acquiring CA information for follower catalog manager: success
F20260806 12:27:29.773729 22253 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ffa5da0d9ad google::LogMessage::Fail() at ??:0
@ 0x7ffa5da1088a google::LogMessage::SendToLog() at ??:0
@ 0x7ffa5da0d4de google::LogMessage::Flush() at ??:0
@ 0x7ffa5da0ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffa606d118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ffa68372989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7ffa6834f023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7ffa6834f54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7ffa68348652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7ffa683479e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7ffa6838e541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ffa5fc50512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffa612946db start_thread at ??:0
@ 0x7ffa5c50b61f clone at ??:0
W20260806 12:27:29.871054 22226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:29.871477 22226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:29.871533 22226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:29.876150 22226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:29.876253 22226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:29.876325 22226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:27:29.882164 22226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:34221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=33697
--tserver_master_addrs=127.28.210.126:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:29.883412 22226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:29.884572 22226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:29.895216 22284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:29.895416 22226 server_base.cc:1034] running on GCE node
W20260806 12:27:29.895200 22278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:29.895200 22279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:29.896245 22226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:29.897040 22226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:29.898332 22226 hybrid_clock.cc:648] HybridClock initialized: now 1786019249898297 us; error 59 us; skew 500 ppm
Aug 06 12:27:29 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019249, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:29.901659 22226 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:27:29.902674 22226 webserver.cc:460] Webserver started at http://127.28.210.65:33697/ using document root <none> and password file <none>
I20260806 12:27:29.903420 22226 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:29.903532 22226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:29.908272 22226 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:29.910907 22297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:29.912554 22226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:29.912699 22226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "fc8ef18fff6b43e6b2422e21212148a3"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "5a24654cf2364cf6a88a39b77ab94679"
server_key_iv: "405768457ada9e32ac963f66507d5140"
server_key_version: "encryptionkey@0"
I20260806 12:27:29.913417 22226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:29.956844 22226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:29.960983 22226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:29.961202 22226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:29.961905 22226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:29.963269 22306 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 12:27:29.969720 22226 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 12:27:29.969837 22226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260806 12:27:29.969920 22226 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 12:27:29.972590 22226 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 12:27:29.972673 22226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260806 12:27:29.973181 22306 tablet_bootstrap.cc:492] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3: Bootstrap starting.
I20260806 12:27:29.984666 22226 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:34221
I20260806 12:27:29.986109 22226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:27:29.987010 22482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:34221 every 8 connection(s)
I20260806 12:27:29.995002 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 22226
I20260806 12:27:29.995757 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:34421
--local_ip_for_outbound_sockets=127.28.210.66
--tserver_master_addrs=127.28.210.126:34979
--webserver_port=32987
--webserver_interface=127.28.210.66
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:27:29.997404 22306 log.cc:824] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:30.017060 22306 tablet_bootstrap.cc:492] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 12:27:30.018075 22306 tablet_bootstrap.cc:492] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3: Bootstrap complete.
I20260806 12:27:30.020903 22306 ts_tablet_manager.cc:1397] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.023s
W20260806 12:27:30.027352 22486 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:27:29.989106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:48231)
0806 12:27:29.989840 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:29.989905 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:29.991058 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:30.025957 (+ 34899us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":235,"threads_started":1}
I20260806 12:27:30.026489 22306 raft_consensus.cc:348] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:30.027555 22306 raft_consensus.cc:729] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8ef18fff6b43e6b2422e21212148a3, State: Initialized, Role: FOLLOWER
W20260806 12:27:30.028244 22483 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:34979 (0 consecutive failures): Network error: Failed to ping master at 127.28.210.126:34979: Client connection negotiation failed: client connection to 127.28.210.126:34979: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 12:27:30.028401 22306 consensus_queue.cc:260] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:30.029313 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:30.028915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:35919)
0806 12:27:30.029139 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:30.029223 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260806 12:27:30.029824 22306 ts_tablet_manager.cc:1428] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260806 12:27:30.030292 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:30.029992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:38947)
0806 12:27:30.030145 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:30.030212 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 12:27:30.030515 22483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 12:27:30.031141 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:30.030896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:50667)
0806 12:27:30.031013 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:30.031074 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 12:27:30.138413 22488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:30.138780 22488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:30.138831 22488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:30.143432 22488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:30.143543 22488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:30.143608 22488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:27:30.149736 22488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:34421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=32987
--tserver_master_addrs=127.28.210.126:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:30.151057 22488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:30.152738 22488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:30.166435 22504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:30.166406 22508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:30.166409 22505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:30.166865 22488 server_base.cc:1034] running on GCE node
I20260806 12:27:30.167519 22488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:30.168485 22488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:30.169770 22488 hybrid_clock.cc:648] HybridClock initialized: now 1786019250169721 us; error 69 us; skew 500 ppm
Aug 06 12:27:30 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019250, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:30.173492 22488 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:27:30.174659 22488 webserver.cc:460] Webserver started at http://127.28.210.66:32987/ using document root <none> and password file <none>
I20260806 12:27:30.175328 22488 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:30.175414 22488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:30.180050 22488 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:30.182771 22515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:30.184417 22488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:30.184553 22488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "75556d2cb84a4e24a80c2fc3015370bc"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "934df17d8661aba8757994bbbb2bc668"
server_key_iv: "ef8219566890a06c949d555498453c78"
server_key_version: "encryptionkey@0"
I20260806 12:27:30.185204 22488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:30.199412 22488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:30.203243 22488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:30.203485 22488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:30.204243 22488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:30.206738 22488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:30.206808 22488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:30.206877 22488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:30.206909 22488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:30.218183 22488 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:34421
I20260806 12:27:30.218297 22695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:34421 every 8 connection(s)
I20260806 12:27:30.219746 22488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:27:30.225836 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:30.223230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:36833)
0806 12:27:30.224012 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:30.224217 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":211,"threads_started":1}
W20260806 12:27:30.226564 22697 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:34979 (0 consecutive failures): Network error: Failed to ping master at 127.28.210.126:34979: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
I20260806 12:27:30.227435 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:30.227121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:43731)
0806 12:27:30.227290 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:30.227356 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 12:27:30.228169 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 22488
I20260806 12:27:30.228282 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:30.227989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:56459)
0806 12:27:30.228144 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:30.228206 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 12:27:30.228500 22697 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 12:27:30.228718 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:36601
--local_ip_for_outbound_sockets=127.28.210.67
--tserver_master_addrs=127.28.210.126:34979
--webserver_port=37213
--webserver_interface=127.28.210.67
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:30.364987 22700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:30.365386 22700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:30.365442 22700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:30.370240 22700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:30.370363 22700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:30.370434 22700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:27:30.376861 22700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:36601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=37213
--tserver_master_addrs=127.28.210.126:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:30.378355 22700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:30.379630 22700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:30.391601 22706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:30.391601 22708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:30.391700 22700 server_base.cc:1034] running on GCE node
W20260806 12:27:30.391601 22710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:30.392671 22700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:30.393589 22700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:30.394799 22700 hybrid_clock.cc:648] HybridClock initialized: now 1786019250394777 us; error 59 us; skew 500 ppm
Aug 06 12:27:30 dist-test-slave-lgr5 krb5kdc[20514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019250, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:30.398974 22700 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:27:30.400367 22700 webserver.cc:460] Webserver started at http://127.28.210.67:37213/ using document root <none> and password file <none>
I20260806 12:27:30.401007 22700 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:30.401082 22700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:30.405524 22700 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 12:27:30.408031 22722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:30.409624 22700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 12:27:30.409775 22700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "2a29258d81234ab29370538ce0b88904"
format_stamp: "Formatted at 2026-08-06 12:26:47 on dist-test-slave-lgr5"
server_key: "0fcad59ac18352cd2a60add840b1d254"
server_key_iv: "31dec7b0e796a1d2f972258060b3408d"
server_key_version: "encryptionkey@0"
I20260806 12:27:30.410663 22700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:30.438617 22700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:30.442415 22700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:30.442623 22700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:30.443312 22700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:30.444497 22700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:30.444571 22700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:30.444622 22700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:30.444650 22700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:30.455638 22700 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:36601
I20260806 12:27:30.455667 22906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:36601 every 8 connection(s)
I20260806 12:27:30.458062 22700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:27:30.459277 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 22700
I20260806 12:27:30.462893 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:30.460885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:50115)
0806 12:27:30.461434 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:30.461563 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":126,"threads_started":1}
W20260806 12:27:30.463490 22907 heartbeater.cc:644] Failed to heartbeat to 127.28.210.126:34979 (0 consecutive failures): Network error: Failed to ping master at 127.28.210.126:34979: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
I20260806 12:27:30.464340 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:30.463980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:57793)
0806 12:27:30.464180 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:30.464240 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 12:27:30.465010 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:30.464769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:41349)
0806 12:27:30.464888 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:30.464947 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 12:27:30.465198 22907 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 12:27:31.032711 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:31.032205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:55505)
0806 12:27:31.032513 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:31.032586 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260806 12:27:31.175465 22913 raft_consensus.cc:482] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:27:31.175698 22913 raft_consensus.cc:504] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:31.177026 22913 leader_election.cc:288] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421), 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601)
Aug 06 12:27:31 dist-test-slave-lgr5 krb5kdc[20514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019249, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.66@KRBTEST.COM
Aug 06 12:27:31 dist-test-slave-lgr5 krb5kdc[20514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019249, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.67@KRBTEST.COM
I20260806 12:27:31.193921 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:31.177335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:34421 (local address 127.28.210.65:52971)
0806 12:27:31.177503 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:31.177521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:31.177657 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:31.179635 (+ 1978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:31.179641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:31.180250 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:31.180852 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:31.180870 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:31.182305 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:31.182308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:31.183423 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:31.183436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:31.184301 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:31.184305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:31.184544 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:31.185301 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:31.185326 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:31.187425 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:31.190009 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:31.190013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:31.190015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:31.190628 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:31.191161 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:31.191165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:31.191167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:31.191462 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:31.192464 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:31.192473 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:31.192906 (+ 433us) client_negotiation.cc:769] Sending connection context
0806 12:27:31.193177 (+ 271us) client_negotiation.cc:240] Negotiation successful
0806 12:27:31.193673 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":333,"spinlock_wait_cycles":158848}
I20260806 12:27:31.193921 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:31.177335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:36601 (local address 127.28.210.65:36563)
0806 12:27:31.177895 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:31.177912 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:31.178036 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:31.179634 (+ 1598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:31.179641 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:31.180251 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:31.180850 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:31.180870 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:31.182305 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:31.182308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:31.183425 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:31.183438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:31.184301 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:31.184305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:31.184545 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:31.185284 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:31.185308 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:31.187705 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:31.189980 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:31.189989 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:31.190007 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:31.190628 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:31.191161 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:31.191165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:31.191167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:31.191462 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:31.192464 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:31.192473 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:31.192906 (+ 433us) client_negotiation.cc:769] Sending connection context
0806 12:27:31.193176 (+ 270us) client_negotiation.cc:240] Negotiation successful
0806 12:27:31.193744 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":544,"spinlock_wait_cycles":22144,"thread_start_us":81,"threads_started":1}
I20260806 12:27:31.193935 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:31.177501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:52971 (local address 127.28.210.66:34421)
0806 12:27:31.177952 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:31.177969 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:31.178047 (+ 78us) server_negotiation.cc:407] Connection header received
0806 12:27:31.178748 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:31.178755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:31.179172 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:31.179505 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:31.181060 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:31.182118 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:31.183599 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:31.184162 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:31.184327 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:31.187722 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:31.187728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:31.187744 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:31.187781 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:31.189757 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:31.190801 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:31.190809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:31.190816 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:31.191038 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:31.191575 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:31.191583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:31.191585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:31.192289 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:31.192447 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:31.193336 (+ 889us) server_negotiation.cc:299] Negotiation successful
0806 12:27:31.193708 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":102,"threads_started":1}
I20260806 12:27:31.193928 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:31.177714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:36563 (local address 127.28.210.67:36601)
0806 12:27:31.178082 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:31.178098 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:31.178142 (+ 44us) server_negotiation.cc:407] Connection header received
0806 12:27:31.178851 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:31.178858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:31.179214 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:31.179509 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:31.181061 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:31.182119 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:31.183597 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:31.184163 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:31.184327 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:31.187961 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:31.187967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:31.187981 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:31.188018 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:31.189865 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:31.190802 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:31.190809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:31.190816 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:31.191038 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:31.191571 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:31.191575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:31.191577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:31.192289 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:31.192447 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:31.193336 (+ 889us) server_negotiation.cc:299] Negotiation successful
0806 12:27:31.193708 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":117,"threads_started":1}
W20260806 12:27:31.197357 22298 leader_election.cc:341] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601): Not found: Tablet not found: 18b26e0d9f754f8eb2b882e118a8bba0
W20260806 12:27:31.197826 22301 leader_election.cc:341] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421): Not found: Tablet not found: 18b26e0d9f754f8eb2b882e118a8bba0
I20260806 12:27:31.197932 22301 leader_election.cc:302] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc8ef18fff6b43e6b2422e21212148a3; no voters: 2a29258d81234ab29370538ce0b88904, 75556d2cb84a4e24a80c2fc3015370bc
I20260806 12:27:31.198266 22913 raft_consensus.cc:2726] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 12:27:31.229875 22918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:31.229330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:46013)
0806 12:27:31.229673 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:31.229748 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
I20260806 12:27:31.467012 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:31.466220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:35463)
0806 12:27:31.466766 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:31.466863 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":157,"threads_started":1}
I20260806 12:27:32.034518 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:32.033839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:39527)
0806 12:27:32.034325 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:32.034402 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":117,"threads_started":1}
I20260806 12:27:32.231513 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:32.230985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:59001)
0806 12:27:32.231285 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:32.231350 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
I20260806 12:27:32.468519 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:32.467988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:51945)
0806 12:27:32.468311 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:32.468383 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":115,"threads_started":1}
I20260806 12:27:32.927223 22926 raft_consensus.cc:482] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:27:32.927402 22926 raft_consensus.cc:504] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:32.927875 22926 leader_election.cc:288] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421), 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601)
W20260806 12:27:32.928762 22298 leader_election.cc:341] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601): Not found: Tablet not found: 18b26e0d9f754f8eb2b882e118a8bba0
W20260806 12:27:32.928983 22301 leader_election.cc:341] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421): Not found: Tablet not found: 18b26e0d9f754f8eb2b882e118a8bba0
I20260806 12:27:32.929199 22301 leader_election.cc:302] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc8ef18fff6b43e6b2422e21212148a3; no voters: 2a29258d81234ab29370538ce0b88904, 75556d2cb84a4e24a80c2fc3015370bc
I20260806 12:27:32.929414 22926 raft_consensus.cc:2726] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 12:27:33.036387 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:33.035755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:34877)
0806 12:27:33.036176 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:33.036251 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":107,"threads_started":1}
I20260806 12:27:33.233213 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:33.232664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:49707)
0806 12:27:33.233024 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:33.233100 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":96,"threads_started":1}
I20260806 12:27:33.469823 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:33.469368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:44427)
0806 12:27:33.469662 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:33.469720 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
I20260806 12:27:34.038035 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:34.037421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:59547)
0806 12:27:34.037826 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:34.037923 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":87,"threads_started":1}
I20260806 12:27:34.234906 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:34.234242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:60017)
0806 12:27:34.234669 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:34.234780 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":116,"threads_started":1}
I20260806 12:27:34.471517 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:34.470890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:44845)
0806 12:27:34.471281 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:34.471388 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260806 12:27:35.040763 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:35.039933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:48085)
0806 12:27:35.040549 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:35.040645 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":319,"threads_started":1}
I20260806 12:27:35.236645 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:35.236131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:32905)
0806 12:27:35.236448 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:35.236532 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":90,"threads_started":1}
I20260806 12:27:35.473994 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:35.472962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:54817)
0806 12:27:35.473463 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:35.473712 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":100,"threads_started":1}
I20260806 12:27:36.027333 22942 raft_consensus.cc:482] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:27:36.027486 22942 raft_consensus.cc:504] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:36.028004 22942 leader_election.cc:288] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421), 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601)
W20260806 12:27:36.028936 22301 leader_election.cc:341] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421): Not found: Tablet not found: 18b26e0d9f754f8eb2b882e118a8bba0
W20260806 12:27:36.029053 22298 leader_election.cc:341] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601): Not found: Tablet not found: 18b26e0d9f754f8eb2b882e118a8bba0
I20260806 12:27:36.029150 22298 leader_election.cc:302] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc8ef18fff6b43e6b2422e21212148a3; no voters: 2a29258d81234ab29370538ce0b88904, 75556d2cb84a4e24a80c2fc3015370bc
I20260806 12:27:36.029367 22942 raft_consensus.cc:2726] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 12:27:36.042430 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:36.041801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:37769)
0806 12:27:36.042198 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:36.042274 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":181,"threads_started":1}
I20260806 12:27:36.238231 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:36.237651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:34231)
0806 12:27:36.238036 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:36.238116 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":102,"threads_started":1}
I20260806 12:27:36.475972 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:36.475337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:41987)
0806 12:27:36.475773 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:36.475855 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":137,"threads_started":1}
I20260806 12:27:37.044211 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:37.043598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:53315)
0806 12:27:37.043977 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:37.044054 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":173,"threads_started":1}
I20260806 12:27:37.239733 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:37.239179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:59631)
0806 12:27:37.239562 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:37.239625 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":110,"threads_started":1}
I20260806 12:27:37.477492 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:37.476929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:35443)
0806 12:27:37.477279 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:37.477380 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":82,"threads_started":1}
I20260806 12:27:37.739521 22961 raft_consensus.cc:482] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:27:37.739682 22961 raft_consensus.cc:504] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:37.740356 22961 leader_election.cc:288] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421), 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601)
W20260806 12:27:37.741243 22301 leader_election.cc:341] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421): Not found: Tablet not found: 18b26e0d9f754f8eb2b882e118a8bba0
W20260806 12:27:37.741354 22298 leader_election.cc:341] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601): Not found: Tablet not found: 18b26e0d9f754f8eb2b882e118a8bba0
I20260806 12:27:37.741420 22298 leader_election.cc:302] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc8ef18fff6b43e6b2422e21212148a3; no voters: 2a29258d81234ab29370538ce0b88904, 75556d2cb84a4e24a80c2fc3015370bc
I20260806 12:27:37.741564 22961 raft_consensus.cc:2726] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 12:27:38.045652 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:38.045098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:40037)
0806 12:27:38.045452 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:38.045536 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":100,"threads_started":1}
I20260806 12:27:38.241331 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:38.240772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:55339)
0806 12:27:38.241110 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:38.241183 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":83,"threads_started":1}
I20260806 12:27:38.479244 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:38.478629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:42921)
0806 12:27:38.479035 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:38.479130 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":105,"threads_started":1}
I20260806 12:27:39.047302 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:39.046773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:48143)
0806 12:27:39.047113 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:39.047185 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":91,"threads_started":1}
I20260806 12:27:39.242782 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:39.242321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:60859)
0806 12:27:39.242602 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:39.242669 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
I20260806 12:27:39.480942 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:39.480374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:41979)
0806 12:27:39.480745 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:39.480826 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260806 12:27:40.049046 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:40.048497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:41141)
0806 12:27:40.048857 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:40.048934 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20260806 12:27:40.244099 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:40.243595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:54057)
0806 12:27:40.243890 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:40.243967 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260806 12:27:40.481864 22976 raft_consensus.cc:482] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:27:40.482052 22976 raft_consensus.cc:504] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75556d2cb84a4e24a80c2fc3015370bc" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34421 } } peers { permanent_uuid: "2a29258d81234ab29370538ce0b88904" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36601 } } peers { permanent_uuid: "fc8ef18fff6b43e6b2422e21212148a3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 34221 } }
I20260806 12:27:40.482539 22976 leader_election.cc:288] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421), 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601)
I20260806 12:27:40.482560 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:40.482017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:54091)
0806 12:27:40.482327 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:40.482403 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
W20260806 12:27:40.483532 22298 leader_election.cc:341] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a29258d81234ab29370538ce0b88904 (127.28.210.67:36601): Not found: Tablet not found: 18b26e0d9f754f8eb2b882e118a8bba0
W20260806 12:27:40.483705 22301 leader_election.cc:341] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75556d2cb84a4e24a80c2fc3015370bc (127.28.210.66:34421): Not found: Tablet not found: 18b26e0d9f754f8eb2b882e118a8bba0
I20260806 12:27:40.483819 22301 leader_election.cc:302] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc8ef18fff6b43e6b2422e21212148a3; no voters: 2a29258d81234ab29370538ce0b88904, 75556d2cb84a4e24a80c2fc3015370bc
I20260806 12:27:40.484014 22976 raft_consensus.cc:2726] T 18b26e0d9f754f8eb2b882e118a8bba0 P fc8ef18fff6b43e6b2422e21212148a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 12:27:41.050767 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:41.050094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:53401)
0806 12:27:41.050511 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:41.050599 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":108,"threads_started":1}
I20260806 12:27:41.245791 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:41.245097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:33795)
0806 12:27:41.245527 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:41.245619 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":132,"threads_started":1}
I20260806 12:27:41.484134 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:41.483558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:59489)
0806 12:27:41.483914 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:41.483984 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260806 12:27:42.052492 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:42.051940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:53659)
0806 12:27:42.052261 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:42.052331 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":91,"threads_started":1}
I20260806 12:27:42.247274 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:42.246787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:41583)
0806 12:27:42.247098 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:42.247165 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
I20260806 12:27:42.485581 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:42.485094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:37039)
0806 12:27:42.485395 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:42.485468 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
I20260806 12:27:43.054035 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:43.053463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:60817)
0806 12:27:43.053844 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:43.053922 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":124,"threads_started":1}
I20260806 12:27:43.248708 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:43.248204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:52587)
0806 12:27:43.248541 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:43.248603 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":84,"threads_started":1}
I20260806 12:27:43.487197 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:43.486647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:53301)
0806 12:27:43.487014 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:43.487086 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":94,"threads_started":1}
I20260806 12:27:44.055595 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:44.055092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:56315)
0806 12:27:44.055397 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:44.055474 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260806 12:27:44.250185 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:44.249665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:39839)
0806 12:27:44.249992 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:44.250071 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":109,"threads_started":1}
I20260806 12:27:44.488804 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:44.488316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:53437)
0806 12:27:44.488621 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:44.488690 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
I20260806 12:27:45.057333 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:45.056831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.65:51927)
0806 12:27:45.057148 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:45.057216 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":85,"threads_started":1}
I20260806 12:27:45.252003 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:45.251410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.66:34723)
0806 12:27:45.251805 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:45.251874 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":325,"threads_started":1}
I20260806 12:27:45.462730 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 22226
I20260806 12:27:45.480568 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 22488
I20260806 12:27:45.490855 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:45.490131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34979 (local address 127.28.210.67:35297)
0806 12:27:45.490613 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:45.490695 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.126:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":141,"threads_started":1}
I20260806 12:27:45.498829 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 22700
2026-08-06T12:27:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (60130 ms)
[----------] 30 tests from SecurityITest (269662 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:27:45 dist-test-slave-lgr5 krb5kdc[23015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:27:45 dist-test-slave-lgr5 krb5kdc[23015](info): set up 2 sockets
Aug 06 12:27:45 dist-test-slave-lgr5 krb5kdc[23015](info): commencing operation
krb5kdc: starting...
W20260806 12:27:46.755734 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:27:46 dist-test-slave-lgr5 krb5kdc[23015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:27:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:27:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:27:46.905376 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:44737
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:44737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:47.040500 23040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:47.040855 23040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:47.040900 23040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:47.045435 23040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:27:47.045528 23040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:47.045548 23040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:47.045565 23040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:27:47.045579 23040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:27:47.052243 23040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:44737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:44737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:47.053778 23040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:47.055162 23040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:47.066499 23045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:47.066690 23050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:47.066720 23040 server_base.cc:1034] running on GCE node
W20260806 12:27:47.066499 23048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:47.067797 23040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:47.069319 23040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:47.070600 23040 hybrid_clock.cc:648] HybridClock initialized: now 1786019267070570 us; error 48 us; skew 500 ppm
Aug 06 12:27:47 dist-test-slave-lgr5 krb5kdc[23015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019267, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:47.074776 23040 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:27:47.076152 23040 webserver.cc:460] Webserver started at http://127.28.210.126:37991/ using document root <none> and password file <none>
I20260806 12:27:47.076864 23040 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:47.076949 23040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:47.077442 23040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:47.080190 23040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "f1d9e0779c5e4fa294a7458335aa508b"
format_stamp: "Formatted at 2026-08-06 12:27:47 on dist-test-slave-lgr5"
server_key: "c6fd19a5e0ca6ecbeb12f4d371323fb6"
server_key_iv: "3fb0f070f3c21f6e12114feeb6bdb1af"
server_key_version: "encryptionkey@0"
I20260806 12:27:47.081009 23040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "f1d9e0779c5e4fa294a7458335aa508b"
format_stamp: "Formatted at 2026-08-06 12:27:47 on dist-test-slave-lgr5"
server_key: "c6fd19a5e0ca6ecbeb12f4d371323fb6"
server_key_iv: "3fb0f070f3c21f6e12114feeb6bdb1af"
server_key_version: "encryptionkey@0"
I20260806 12:27:47.086483 23040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 12:27:47.090508 23061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:47.093317 23040 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 12:27:47.093545 23040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "f1d9e0779c5e4fa294a7458335aa508b"
format_stamp: "Formatted at 2026-08-06 12:27:47 on dist-test-slave-lgr5"
server_key: "c6fd19a5e0ca6ecbeb12f4d371323fb6"
server_key_iv: "3fb0f070f3c21f6e12114feeb6bdb1af"
server_key_version: "encryptionkey@0"
I20260806 12:27:47.093739 23040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:47.114874 23040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:47.119906 23040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:47.120219 23040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:47.128827 23040 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:44737
I20260806 12:27:47.128872 23146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:44737 every 8 connection(s)
I20260806 12:27:47.129952 23040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:27:47.133256 23148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:47.135848 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 23040
I20260806 12:27:47.136030 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:27:47.136413 29513 external_mini_cluster.cc:1428] Setting key ecd7338fcae044e1c138def95b18159c
I20260806 12:27:47.141551 23148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b: Bootstrap starting.
I20260806 12:27:47.144287 23148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:47.145275 23148 log.cc:824] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b: Log is configured to *not* fsync() on all Append() calls
Aug 06 12:27:47 dist-test-slave-lgr5 krb5kdc[23015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:27:47.148367 23148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b: No bootstrap required, opened a new log
I20260806 12:27:47.153934 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:47.138247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.28.210.126:44737)
0806 12:27:47.138817 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:47.138834 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:47.138886 (+ 52us) server_negotiation.cc:407] Connection header received
0806 12:27:47.139902 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:47.139925 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:47.140398 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:47.140758 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:47.141669 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:47.142764 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:47.143658 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:47.144247 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:47.144411 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:47.147549 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:47.147557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:47.147579 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:47.147621 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:47.149872 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:47.150465 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:47.150472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:47.150480 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:47.150651 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:47.151004 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:47.151008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:47.151010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:47.151660 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:47.151886 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:47.152244 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 12:27:47.152742 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":175,"threads_started":1}
I20260806 12:27:47.153765 23148 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d9e0779c5e4fa294a7458335aa508b" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44737 } }
I20260806 12:27:47.154093 23148 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:47.154145 23148 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d9e0779c5e4fa294a7458335aa508b, State: Initialized, Role: FOLLOWER
I20260806 12:27:47.154675 23148 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d9e0779c5e4fa294a7458335aa508b" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44737 } }
I20260806 12:27:47.154850 23148 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:27:47.154984 23148 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:27:47.155120 23148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:47.156317 23148 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d9e0779c5e4fa294a7458335aa508b" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44737 } }
I20260806 12:27:47.156692 23148 leader_election.cc:302] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1d9e0779c5e4fa294a7458335aa508b; no voters:
I20260806 12:27:47.157048 23148 leader_election.cc:288] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:27:47.157164 23154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:47.157424 23154 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [term 1 LEADER]: Becoming Leader. State: Replica: f1d9e0779c5e4fa294a7458335aa508b, State: Running, Role: LEADER
I20260806 12:27:47.157783 23154 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d9e0779c5e4fa294a7458335aa508b" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44737 } }
I20260806 12:27:47.158362 23148 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:27:47.159600 23161 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1d9e0779c5e4fa294a7458335aa508b. Latest consensus state: current_term: 1 leader_uuid: "f1d9e0779c5e4fa294a7458335aa508b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d9e0779c5e4fa294a7458335aa508b" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44737 } } }
I20260806 12:27:47.159744 23161 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:47.160053 23158 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1d9e0779c5e4fa294a7458335aa508b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1d9e0779c5e4fa294a7458335aa508b" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44737 } } }
I20260806 12:27:47.160180 23158 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:47.160208 23176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:27:47.164378 23176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:27:47.171406 23176 catalog_manager.cc:1347] Generated new cluster ID: 8327b29332774ffd9f46cbbf47dc9f90
I20260806 12:27:47.171527 23176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:27:47.185814 23176 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:27:47.186875 23176 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:27:47.194615 23176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b: Generated new TSK 0
I20260806 12:27:47.195785 23176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:27:47.236277 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:44737
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:47.380661 23204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:47.381036 23204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:47.381094 23204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:47.387051 23204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:47.387171 23204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:47.387241 23204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:27:47.393606 23204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:44737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:47.395149 23204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:47.396607 23204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:47.408891 23213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:47.409202 23204 server_base.cc:1034] running on GCE node
W20260806 12:27:47.408896 23215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:47.408893 23212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:47.409917 23204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:47.410714 23204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:47.411926 23204 hybrid_clock.cc:648] HybridClock initialized: now 1786019267411891 us; error 47 us; skew 500 ppm
Aug 06 12:27:47 dist-test-slave-lgr5 krb5kdc[23015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019267, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:47.421097 23204 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:27:47.422492 23204 webserver.cc:460] Webserver started at http://127.28.210.65:41511/ using document root <none> and password file <none>
I20260806 12:27:47.423189 23204 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:47.423273 23204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:47.423532 23204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:47.426133 23204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9"
format_stamp: "Formatted at 2026-08-06 12:27:47 on dist-test-slave-lgr5"
server_key: "71efa4ba27adc3c93506f86898382200"
server_key_iv: "aed23d09d3e931c6939da4d748d02c50"
server_key_version: "encryptionkey@0"
I20260806 12:27:47.427037 23204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9"
format_stamp: "Formatted at 2026-08-06 12:27:47 on dist-test-slave-lgr5"
server_key: "71efa4ba27adc3c93506f86898382200"
server_key_iv: "aed23d09d3e931c6939da4d748d02c50"
server_key_version: "encryptionkey@0"
I20260806 12:27:47.432799 23204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 12:27:47.436223 23223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:47.438119 23204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:27:47.438318 23204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9"
format_stamp: "Formatted at 2026-08-06 12:27:47 on dist-test-slave-lgr5"
server_key: "71efa4ba27adc3c93506f86898382200"
server_key_iv: "aed23d09d3e931c6939da4d748d02c50"
server_key_version: "encryptionkey@0"
I20260806 12:27:47.438484 23204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:47.455945 23204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:47.460453 23204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:47.460695 23204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:47.461558 23204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:47.462575 23204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:47.462634 23204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:47.462682 23204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:47.462706 23204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:47.473479 23204 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:36735
I20260806 12:27:47.473556 23416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:36735 every 8 connection(s)
I20260806 12:27:47.474622 23204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:27:47.477294 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 23204
I20260806 12:27:47.477460 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:27:47.477837 29513 external_mini_cluster.cc:1428] Setting key 5bc58e900d87e9e31f2cd242b212082a
Aug 06 12:27:47 dist-test-slave-lgr5 krb5kdc[23015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019267, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:27:47.492375 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:47.476876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:43847 (local address 127.28.210.126:44737)
0806 12:27:47.477037 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:47.477041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:47.478377 (+ 1336us) server_negotiation.cc:407] Connection header received
0806 12:27:47.480197 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:47.480201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:47.480267 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:47.480379 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:47.482029 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:47.482744 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:47.484216 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:47.484472 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:47.484578 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:47.487969 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:47.487973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:47.487976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:47.488010 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:47.489620 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:47.490298 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:47.490304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:47.490306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:47.490406 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:47.490880 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:47.490884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:47.490886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:47.491122 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:47.491225 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:47.491961 (+ 736us) server_negotiation.cc:299] Negotiation successful
0806 12:27:47.492170 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 12:27:47.493614 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:47.477450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:44737 (local address 127.28.210.65:43847)
0806 12:27:47.478160 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:47.478219 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:47.479863 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:47.480742 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:47.480751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:47.481191 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:47.481826 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:47.481852 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:47.482944 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:47.482947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:47.484036 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:47.484051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:47.484662 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:47.484666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:47.484917 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:47.485655 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:47.485677 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:47.487693 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:47.489793 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:47.489802 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:47.489826 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:47.490172 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:47.490516 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:47.490519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:47.490522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:47.490769 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:47.491247 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:47.491257 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:47.491651 (+ 394us) client_negotiation.cc:769] Sending connection context
0806 12:27:47.491893 (+ 242us) client_negotiation.cc:240] Negotiation successful
0806 12:27:47.492385 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":178,"threads_started":1}
I20260806 12:27:47.495482 23418 heartbeater.cc:347] Connected to a master server at 127.28.210.126:44737
I20260806 12:27:47.495810 23418 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:47.496508 23418 heartbeater.cc:511] Master 127.28.210.126:44737 requested a full tablet report, sending...
I20260806 12:27:47.498544 23083 ts_manager.cc:194] Registered new tserver with Master: 1bd90b4e8a8e44269f4ad12d8fb0c1d9 (127.28.210.65:36735)
I20260806 12:27:47.499930 23083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:43847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:27:47.512847 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:44737
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:47.665956 23428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:47.666256 23428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:47.666303 23428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:47.670552 23428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:47.670646 23428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:47.670709 23428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:27:47.677037 23428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:44737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:47.678350 23428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:47.679579 23428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:47.692112 23436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:47.692112 23441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:47.692427 23437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:47.692492 23428 server_base.cc:1034] running on GCE node
I20260806 12:27:47.693056 23428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:47.693854 23428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:47.695065 23428 hybrid_clock.cc:648] HybridClock initialized: now 1786019267695024 us; error 50 us; skew 500 ppm
Aug 06 12:27:47 dist-test-slave-lgr5 krb5kdc[23015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019267, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:47.698678 23428 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:27:47.699831 23428 webserver.cc:460] Webserver started at http://127.28.210.66:42463/ using document root <none> and password file <none>
I20260806 12:27:47.700541 23428 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:47.700625 23428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:47.700865 23428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:47.703239 23428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "b893095cd36e4fd9910183c998cfffd6"
format_stamp: "Formatted at 2026-08-06 12:27:47 on dist-test-slave-lgr5"
server_key: "dc755f3307361af344ecf6f671b6ab70"
server_key_iv: "67b878311bce58289a2924c6705b64de"
server_key_version: "encryptionkey@0"
I20260806 12:27:47.704125 23428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "b893095cd36e4fd9910183c998cfffd6"
format_stamp: "Formatted at 2026-08-06 12:27:47 on dist-test-slave-lgr5"
server_key: "dc755f3307361af344ecf6f671b6ab70"
server_key_iv: "67b878311bce58289a2924c6705b64de"
server_key_version: "encryptionkey@0"
I20260806 12:27:47.708869 23428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 12:27:47.711927 23449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:47.713469 23428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 12:27:47.713613 23428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "b893095cd36e4fd9910183c998cfffd6"
format_stamp: "Formatted at 2026-08-06 12:27:47 on dist-test-slave-lgr5"
server_key: "dc755f3307361af344ecf6f671b6ab70"
server_key_iv: "67b878311bce58289a2924c6705b64de"
server_key_version: "encryptionkey@0"
I20260806 12:27:47.713757 23428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:47.734028 23428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:47.738211 23428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:47.738499 23428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:47.739295 23428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:47.740490 23428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:47.740549 23428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:47.740602 23428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:47.740617 23428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:47.752018 23428 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:36915
I20260806 12:27:47.752184 23621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:36915 every 8 connection(s)
I20260806 12:27:47.753264 23428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:27:47.754616 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 23428
I20260806 12:27:47.754757 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:27:47.755103 29513 external_mini_cluster.cc:1428] Setting key f65f75192d1c30d96ec6dcdc5b9c815a
Aug 06 12:27:47 dist-test-slave-lgr5 krb5kdc[23015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019267, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
I20260806 12:27:47.770058 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:47.755826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:51441 (local address 127.28.210.126:44737)
0806 12:27:47.755971 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:47.755976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:47.757184 (+ 1208us) server_negotiation.cc:407] Connection header received
0806 12:27:47.758694 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:47.758697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:47.758762 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:47.758871 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:47.760534 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:47.761154 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:47.762626 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:47.762864 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:47.762966 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:47.765846 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:47.765850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:47.765853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:47.765886 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:47.767376 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:47.767981 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:47.767986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:47.767988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:47.768118 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:47.768594 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:47.768597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:47.768598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:47.768862 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:47.768951 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:47.769710 (+ 759us) server_negotiation.cc:299] Negotiation successful
0806 12:27:47.769870 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 12:27:47.771019 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:47.756367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:44737 (local address 127.28.210.66:51441)
0806 12:27:47.757030 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:47.757084 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:47.758450 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:47.759245 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:47.759254 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:47.759689 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:47.760366 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:47.760388 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:47.761320 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:47.761323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:47.762485 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:47.762499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:47.763042 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:47.763047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:47.763238 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:47.763894 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:47.763915 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:47.765611 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:47.767521 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:47.767529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:47.767550 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:47.767857 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:47.768244 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:47.768249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:47.768251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:47.768469 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:47.769014 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:47.769023 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:47.769397 (+ 374us) client_negotiation.cc:769] Sending connection context
0806 12:27:47.769613 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 12:27:47.770039 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":186,"threads_started":1}
I20260806 12:27:47.772603 23622 heartbeater.cc:347] Connected to a master server at 127.28.210.126:44737
I20260806 12:27:47.773039 23622 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:47.773720 23622 heartbeater.cc:511] Master 127.28.210.126:44737 requested a full tablet report, sending...
I20260806 12:27:47.775277 23083 ts_manager.cc:194] Registered new tserver with Master: b893095cd36e4fd9910183c998cfffd6 (127.28.210.66:36915)
I20260806 12:27:47.776244 23083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:51441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:27:47.790504 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:44737
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:47.917014 23629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:47.917335 23629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:47.917385 23629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:47.921921 23629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:47.922012 23629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:47.922083 23629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:27:47.927884 23629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:44737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:47.929159 23629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:47.930297 23629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:47.940997 23644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:47.941124 23629 server_base.cc:1034] running on GCE node
W20260806 12:27:47.941033 23645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:47.941030 23647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:47.941811 23629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:47.942545 23629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:47.943717 23629 hybrid_clock.cc:648] HybridClock initialized: now 1786019267943700 us; error 29 us; skew 500 ppm
Aug 06 12:27:47 dist-test-slave-lgr5 krb5kdc[23015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019267, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:47.947285 23629 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:27:47.948530 23629 webserver.cc:460] Webserver started at http://127.28.210.67:43513/ using document root <none> and password file <none>
I20260806 12:27:47.949337 23629 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:47.949424 23629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:47.949669 23629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:47.952152 23629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "db94adced85e426c8b020c95ff95deca"
format_stamp: "Formatted at 2026-08-06 12:27:47 on dist-test-slave-lgr5"
server_key: "a6365d6f9e63aeafa05f7afe48fbd742"
server_key_iv: "23af4edf0be35dbc0ced78edce1f27ec"
server_key_version: "encryptionkey@0"
I20260806 12:27:47.952834 23629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "db94adced85e426c8b020c95ff95deca"
format_stamp: "Formatted at 2026-08-06 12:27:47 on dist-test-slave-lgr5"
server_key: "a6365d6f9e63aeafa05f7afe48fbd742"
server_key_iv: "23af4edf0be35dbc0ced78edce1f27ec"
server_key_version: "encryptionkey@0"
I20260806 12:27:47.957137 23629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:27:47.960438 23655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:47.962148 23629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:47.962303 23629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "db94adced85e426c8b020c95ff95deca"
format_stamp: "Formatted at 2026-08-06 12:27:47 on dist-test-slave-lgr5"
server_key: "a6365d6f9e63aeafa05f7afe48fbd742"
server_key_iv: "23af4edf0be35dbc0ced78edce1f27ec"
server_key_version: "encryptionkey@0"
I20260806 12:27:47.962453 23629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:48.002458 23629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:48.006811 23629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:48.007046 23629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:48.007925 23629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:48.009014 23629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:48.009068 23629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:48.009112 23629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:48.009140 23629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:48.021641 23629 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:36723
I20260806 12:27:48.021718 23853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:36723 every 8 connection(s)
I20260806 12:27:48.022866 23629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:27:48.030717 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 23629
I20260806 12:27:48.030892 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:27:48.031216 29513 external_mini_cluster.cc:1428] Setting key 8c1c7745b44984858a7550d462d1fd68
Aug 06 12:27:48 dist-test-slave-lgr5 krb5kdc[23015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019267, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:27:48.041383 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.025363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:48905 (local address 127.28.210.126:44737)
0806 12:27:48.025564 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:48.025569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:48.026692 (+ 1123us) server_negotiation.cc:407] Connection header received
0806 12:27:48.028465 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:48.028469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:48.028541 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:48.028625 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:48.030132 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.030871 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.032479 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.032854 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.032970 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.036438 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:48.036444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:48.036447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:48.036493 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:48.038203 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:48.039016 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:48.039020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:48.039022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:48.039123 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:48.039661 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:48.039665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:48.039666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:48.039968 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:48.040059 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:48.040921 (+ 862us) server_negotiation.cc:299] Negotiation successful
0806 12:27:48.041211 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 12:27:48.042353 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.025834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:44737 (local address 127.28.210.67:48905)
0806 12:27:48.026492 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:48.026551 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:48.028118 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:48.028888 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:48.028896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:48.029330 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:48.029918 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.029935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.031047 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.031050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.032351 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.032364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.033050 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.033054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.033287 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.034138 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:48.034233 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:48.036206 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:48.038413 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:48.038425 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:48.038449 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:48.038807 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:48.039281 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:48.039286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:48.039293 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:48.039540 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:48.040196 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:48.040213 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:48.040574 (+ 361us) client_negotiation.cc:769] Sending connection context
0806 12:27:48.040769 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 12:27:48.041222 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":170,"threads_started":1}
I20260806 12:27:48.044332 23855 heartbeater.cc:347] Connected to a master server at 127.28.210.126:44737
I20260806 12:27:48.044703 23855 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:48.045614 23855 heartbeater.cc:511] Master 127.28.210.126:44737 requested a full tablet report, sending...
I20260806 12:27:48.047770 23083 ts_manager.cc:194] Registered new tserver with Master: db94adced85e426c8b020c95ff95deca (127.28.210.67:36723)
I20260806 12:27:48.048513 23083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:48905
I20260806 12:27:48.059913 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:27:48.071130 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.062401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59776 (local address 127.28.210.126:44737)
0806 12:27:48.062530 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:48.062535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:48.062662 (+ 127us) server_negotiation.cc:407] Connection header received
0806 12:27:48.062812 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:48.062815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:48.062871 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:48.062973 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:48.063777 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.064366 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.065250 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.065486 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.065574 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.067196 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:48.067200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:48.067204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:48.067243 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:48.068689 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:48.069264 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:48.069267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:48.069269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:48.069381 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:48.069991 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:48.069995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:48.069996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:48.070296 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:48.070435 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:48.070833 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 12:27:48.070992 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 12:27:48.075184 23083 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:27:48.078984 23083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:27:48.096704 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.090773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:36735 (local address 127.0.0.1:37292)
0806 12:27:48.091509 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:48.091522 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:48.091604 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:48.092361 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:48.092366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:48.092392 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:48.092506 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.092513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.093588 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.093591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.095117 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.095129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.096044 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.096047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.096205 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.096251 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 12:27:48.096371 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 12:27:48.096521 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":26,"thread_start_us":102,"threads_started":1}
I20260806 12:27:48.096876 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.090402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.28.210.67:36723)
0806 12:27:48.090781 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:48.090795 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:48.090823 (+ 28us) server_negotiation.cc:407] Connection header received
0806 12:27:48.090927 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:48.090932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:48.091542 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:48.092160 (+ 618us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:48.092318 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.093734 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.095293 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.096275 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.096394 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.096539 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:48.096605 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 12:27:48.096708 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":175,"threads_started":1}
I20260806 12:27:48.097249 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.091152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.28.210.65:36735)
0806 12:27:48.091804 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:48.091811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:48.091834 (+ 23us) server_negotiation.cc:407] Connection header received
0806 12:27:48.091908 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:48.091913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:48.092150 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:48.092363 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:48.092653 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.093458 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.095258 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.095910 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.096018 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.096235 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:48.096972 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 12:27:48.097100 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":75,"threads_started":1}
I20260806 12:27:48.096728 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.089912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:36723 (local address 127.0.0.1:60132)
0806 12:27:48.090559 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:48.090619 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:48.090766 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:48.091738 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:48.091742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:48.091782 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:48.091987 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.091995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.093908 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.093912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.095040 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.095052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.096402 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.096405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.096490 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.096511 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 12:27:48.096569 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 12:27:48.096629 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":25,"spinlock_wait_cycles":2304,"thread_start_us":116,"threads_started":1}
I20260806 12:27:48.097807 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.090030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:36915 (local address 127.0.0.1:49902)
0806 12:27:48.091198 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:48.091211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:48.091308 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:48.091734 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:48.091741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:48.091780 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:48.092008 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.092017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.093908 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.093912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.096136 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.096150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.097342 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.097346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.097438 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.097461 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 12:27:48.097578 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 12:27:48.097657 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1103,"mutex_wait_us":130,"spinlock_wait_cycles":11776,"thread_start_us":126,"threads_started":1}
I20260806 12:27:48.098294 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.090861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.28.210.66:36915)
0806 12:27:48.091296 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:48.091309 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:48.091333 (+ 24us) server_negotiation.cc:407] Connection header received
0806 12:27:48.091433 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:48.091438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:48.091593 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:48.091745 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:48.092172 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.093738 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.096313 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.097200 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.097312 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.097457 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:48.098036 (+ 579us) server_negotiation.cc:299] Negotiation successful
0806 12:27:48.098149 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":126,"threads_started":1}
I20260806 12:27:48.100950 23299 tablet_service.cc:1467] Processing CreateTablet for tablet c31ffbc8bd434772914164d925d056f7 (DEFAULT_TABLE table=test-table [id=749cd17593704e2dbb7e9790cf59ddd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:48.102131 23299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31ffbc8bd434772914164d925d056f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:48.102031 23735 tablet_service.cc:1467] Processing CreateTablet for tablet c31ffbc8bd434772914164d925d056f7 (DEFAULT_TABLE table=test-table [id=749cd17593704e2dbb7e9790cf59ddd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:48.102029 23528 tablet_service.cc:1467] Processing CreateTablet for tablet c31ffbc8bd434772914164d925d056f7 (DEFAULT_TABLE table=test-table [id=749cd17593704e2dbb7e9790cf59ddd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:48.103353 23735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31ffbc8bd434772914164d925d056f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:48.103353 23528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31ffbc8bd434772914164d925d056f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:48.110288 23878 tablet_bootstrap.cc:492] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9: Bootstrap starting.
I20260806 12:27:48.112838 23879 tablet_bootstrap.cc:492] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6: Bootstrap starting.
I20260806 12:27:48.113569 23881 tablet_bootstrap.cc:492] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca: Bootstrap starting.
I20260806 12:27:48.114105 23878 tablet_bootstrap.cc:654] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:48.115846 23879 tablet_bootstrap.cc:654] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:48.115829 23878 log.cc:824] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:48.115958 23881 tablet_bootstrap.cc:654] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:48.117460 23881 log.cc:824] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:48.117458 23879 log.cc:824] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:48.118952 23878 tablet_bootstrap.cc:492] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9: No bootstrap required, opened a new log
I20260806 12:27:48.119223 23878 ts_tablet_manager.cc:1397] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 12:27:48.119577 23879 tablet_bootstrap.cc:492] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6: No bootstrap required, opened a new log
I20260806 12:27:48.119737 23881 tablet_bootstrap.cc:492] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca: No bootstrap required, opened a new log
I20260806 12:27:48.119777 23879 ts_tablet_manager.cc:1397] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 12:27:48.119958 23881 ts_tablet_manager.cc:1397] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260806 12:27:48.123284 23879 raft_consensus.cc:348] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db94adced85e426c8b020c95ff95deca" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36723 } } peers { permanent_uuid: "b893095cd36e4fd9910183c998cfffd6" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36915 } } peers { permanent_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36735 } }
I20260806 12:27:48.123595 23879 raft_consensus.cc:374] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:48.123656 23879 raft_consensus.cc:729] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b893095cd36e4fd9910183c998cfffd6, State: Initialized, Role: FOLLOWER
I20260806 12:27:48.124187 23879 consensus_queue.cc:260] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db94adced85e426c8b020c95ff95deca" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36723 } } peers { permanent_uuid: "b893095cd36e4fd9910183c998cfffd6" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36915 } } peers { permanent_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36735 } }
I20260806 12:27:48.124287 23881 raft_consensus.cc:348] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db94adced85e426c8b020c95ff95deca" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36723 } } peers { permanent_uuid: "b893095cd36e4fd9910183c998cfffd6" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36915 } } peers { permanent_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36735 } }
I20260806 12:27:48.124655 23881 raft_consensus.cc:374] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:48.124725 23881 raft_consensus.cc:729] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db94adced85e426c8b020c95ff95deca, State: Initialized, Role: FOLLOWER
I20260806 12:27:48.125164 23622 heartbeater.cc:503] Master 127.28.210.126:44737 was elected leader, sending a full tablet report...
I20260806 12:27:48.125383 23879 ts_tablet_manager.cc:1428] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 12:27:48.125217 23878 raft_consensus.cc:348] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db94adced85e426c8b020c95ff95deca" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36723 } } peers { permanent_uuid: "b893095cd36e4fd9910183c998cfffd6" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36915 } } peers { permanent_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36735 } }
I20260806 12:27:48.125517 23878 raft_consensus.cc:374] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:48.125593 23878 raft_consensus.cc:729] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd90b4e8a8e44269f4ad12d8fb0c1d9, State: Initialized, Role: FOLLOWER
I20260806 12:27:48.126067 23878 consensus_queue.cc:260] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db94adced85e426c8b020c95ff95deca" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36723 } } peers { permanent_uuid: "b893095cd36e4fd9910183c998cfffd6" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36915 } } peers { permanent_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36735 } }
I20260806 12:27:48.126767 23881 consensus_queue.cc:260] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db94adced85e426c8b020c95ff95deca" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36723 } } peers { permanent_uuid: "b893095cd36e4fd9910183c998cfffd6" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36915 } } peers { permanent_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36735 } }
I20260806 12:27:48.127817 23855 heartbeater.cc:503] Master 127.28.210.126:44737 was elected leader, sending a full tablet report...
I20260806 12:27:48.127977 23881 ts_tablet_manager.cc:1428] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 12:27:48.129169 23418 heartbeater.cc:503] Master 127.28.210.126:44737 was elected leader, sending a full tablet report...
I20260806 12:27:48.133626 23878 ts_tablet_manager.cc:1428] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9: Time spent starting tablet: real 0.014s user 0.005s sys 0.004s
I20260806 12:27:48.196069 23890 raft_consensus.cc:482] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:27:48.196345 23890 raft_consensus.cc:504] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db94adced85e426c8b020c95ff95deca" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36723 } } peers { permanent_uuid: "b893095cd36e4fd9910183c998cfffd6" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36915 } } peers { permanent_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36735 } }
I20260806 12:27:48.197749 23890 leader_election.cc:288] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db94adced85e426c8b020c95ff95deca (127.28.210.67:36723), b893095cd36e4fd9910183c998cfffd6 (127.28.210.66:36915)
I20260806 12:27:48.204918 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.198408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:32829 (local address 127.28.210.67:36723)
0806 12:27:48.198607 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:48.198613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:48.198988 (+ 375us) server_negotiation.cc:407] Connection header received
0806 12:27:48.199217 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:48.199220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:48.199291 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:48.199394 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:48.199886 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.200807 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.203041 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.203742 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.203916 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.203986 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:48.204575 (+ 589us) server_negotiation.cc:299] Negotiation successful
0806 12:27:48.204726 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 12:27:48.204918 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.198602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:43365 (local address 127.28.210.66:36915)
0806 12:27:48.198760 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:48.198765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:48.199026 (+ 261us) server_negotiation.cc:407] Connection header received
0806 12:27:48.199214 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:48.199217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:48.199293 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:48.199439 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:48.199876 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.200807 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.203041 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.203738 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.203916 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.203985 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:48.204572 (+ 587us) server_negotiation.cc:299] Negotiation successful
0806 12:27:48.204726 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 12:27:48.205047 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.198451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:36915 (local address 127.28.210.65:43365)
0806 12:27:48.198884 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:48.198896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:48.199061 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:48.199441 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:48.199444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:48.199471 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:48.199742 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.199748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.200994 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.200999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.202711 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.202737 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.203955 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.203961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.204326 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.204390 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 12:27:48.204516 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 12:27:48.204716 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":282,"spinlock_wait_cycles":39808,"thread_start_us":133,"threads_started":1}
I20260806 12:27:48.205488 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.198176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:36723 (local address 127.28.210.65:32829)
0806 12:27:48.198813 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:48.198830 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:48.199061 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:48.199464 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:48.199468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:48.199481 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:48.199732 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.199742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.200994 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.200999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.202711 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.202737 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.203955 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.203961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.204327 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.204390 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 12:27:48.204526 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 12:27:48.204716 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":322,"spinlock_wait_cycles":18304,"thread_start_us":204,"threads_started":1}
I20260806 12:27:48.206032 23558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31ffbc8bd434772914164d925d056f7" candidate_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b893095cd36e4fd9910183c998cfffd6" is_pre_election: true
I20260806 12:27:48.206365 23775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31ffbc8bd434772914164d925d056f7" candidate_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db94adced85e426c8b020c95ff95deca" is_pre_election: true
I20260806 12:27:48.206575 23558 raft_consensus.cc:2454] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bd90b4e8a8e44269f4ad12d8fb0c1d9 in term 0.
I20260806 12:27:48.206663 23775 raft_consensus.cc:2454] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bd90b4e8a8e44269f4ad12d8fb0c1d9 in term 0.
I20260806 12:27:48.207321 23229 leader_election.cc:302] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1bd90b4e8a8e44269f4ad12d8fb0c1d9, b893095cd36e4fd9910183c998cfffd6, db94adced85e426c8b020c95ff95deca; no voters:
I20260806 12:27:48.207667 23890 raft_consensus.cc:2781] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:27:48.207784 23890 raft_consensus.cc:482] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:27:48.207845 23890 raft_consensus.cc:3037] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:48.209200 23890 raft_consensus.cc:504] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db94adced85e426c8b020c95ff95deca" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36723 } } peers { permanent_uuid: "b893095cd36e4fd9910183c998cfffd6" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36915 } } peers { permanent_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36735 } }
I20260806 12:27:48.209728 23890 leader_election.cc:288] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [CANDIDATE]: Term 1 election: Requested vote from peers db94adced85e426c8b020c95ff95deca (127.28.210.67:36723), b893095cd36e4fd9910183c998cfffd6 (127.28.210.66:36915)
I20260806 12:27:48.210278 23775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31ffbc8bd434772914164d925d056f7" candidate_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db94adced85e426c8b020c95ff95deca"
I20260806 12:27:48.210299 23558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31ffbc8bd434772914164d925d056f7" candidate_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b893095cd36e4fd9910183c998cfffd6"
I20260806 12:27:48.210449 23775 raft_consensus.cc:3037] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:48.210453 23558 raft_consensus.cc:3037] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:48.211768 23775 raft_consensus.cc:2454] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bd90b4e8a8e44269f4ad12d8fb0c1d9 in term 1.
I20260806 12:27:48.211803 23558 raft_consensus.cc:2454] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bd90b4e8a8e44269f4ad12d8fb0c1d9 in term 1.
I20260806 12:27:48.212253 23229 leader_election.cc:302] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1bd90b4e8a8e44269f4ad12d8fb0c1d9, b893095cd36e4fd9910183c998cfffd6, db94adced85e426c8b020c95ff95deca; no voters:
I20260806 12:27:48.212478 23890 raft_consensus.cc:2781] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:48.212782 23890 raft_consensus.cc:686] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 1 LEADER]: Becoming Leader. State: Replica: 1bd90b4e8a8e44269f4ad12d8fb0c1d9, State: Running, Role: LEADER
I20260806 12:27:48.213481 23890 consensus_queue.cc:237] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db94adced85e426c8b020c95ff95deca" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36723 } } peers { permanent_uuid: "b893095cd36e4fd9910183c998cfffd6" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36915 } } peers { permanent_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36735 } }
I20260806 12:27:48.218313 23083 catalog_manager.cc:5495] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bd90b4e8a8e44269f4ad12d8fb0c1d9 (127.28.210.65). New cstate: current_term: 1 leader_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db94adced85e426c8b020c95ff95deca" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b893095cd36e4fd9910183c998cfffd6" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bd90b4e8a8e44269f4ad12d8fb0c1d9" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36735 } health_report { overall_health: HEALTHY } } }
W20260806 12:27:48.226370 23421 tablet.cc:2369] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:27:48.254796 23623 tablet.cc:2369] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:27:48.257781 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.252970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.28.210.65:36735)
0806 12:27:48.253112 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:48.253116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:48.253150 (+ 34us) server_negotiation.cc:407] Connection header received
0806 12:27:48.253236 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:48.253238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:48.253307 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:48.253400 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:27:48.253684 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.254330 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.255671 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.255950 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.256213 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.256730 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:27:48.257237 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:27:48.257349 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:48.257520 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 12:27:48.257614 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 12:27:48.265594 23775 raft_consensus.cc:1261] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca [term 1 FOLLOWER]: Refusing update from remote peer 1bd90b4e8a8e44269f4ad12d8fb0c1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:48.265632 23558 raft_consensus.cc:1261] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6 [term 1 FOLLOWER]: Refusing update from remote peer 1bd90b4e8a8e44269f4ad12d8fb0c1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:48.266541 23890 consensus_queue.cc:1035] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db94adced85e426c8b020c95ff95deca" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 36723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:48.266893 23899 consensus_queue.cc:1035] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b893095cd36e4fd9910183c998cfffd6" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 12:27:48.274120 23856 tablet.cc:2369] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:27:48.277278 23903 mvcc.cc:205] Tried to move back new op lower bound from 7315534922805878784 to 7315534922607149056. Current Snapshot: MvccSnapshot[applied={T|T < 7315534922805878784}]
I20260806 12:27:48.278039 23908 mvcc.cc:205] Tried to move back new op lower bound from 7315534922805878784 to 7315534922607149056. Current Snapshot: MvccSnapshot[applied={T|T < 7315534922805878784}]
I20260806 12:27:48.279381 23902 mvcc.cc:205] Tried to move back new op lower bound from 7315534922805878784 to 7315534922607149056. Current Snapshot: MvccSnapshot[applied={T|T < 7315534922805878784 or (T in {7315534922805878784})}]
I20260806 12:27:48.286602 23083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:48.286798 23083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:48.289971 23083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1d9e0779c5e4fa294a7458335aa508b: Sending DeleteTablet for 3 replicas of tablet c31ffbc8bd434772914164d925d056f7
I20260806 12:27:48.291030 23528 tablet_service.cc:1514] Processing DeleteTablet for tablet c31ffbc8bd434772914164d925d056f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:48 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:49902
I20260806 12:27:48.291209 23735 tablet_service.cc:1514] Processing DeleteTablet for tablet c31ffbc8bd434772914164d925d056f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:48 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:60132
I20260806 12:27:48.291604 23299 tablet_service.cc:1514] Processing DeleteTablet for tablet c31ffbc8bd434772914164d925d056f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:48 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:37292
I20260806 12:27:48.291695 23921 tablet_replica.cc:330] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6: stopping tablet replica
I20260806 12:27:48.292116 23921 raft_consensus.cc:2229] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:27:48.292316 23924 tablet_replica.cc:330] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9: stopping tablet replica
I20260806 12:27:48.292567 23921 raft_consensus.cc:2258] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:48.292768 23924 raft_consensus.cc:2229] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:27:48.293037 23922 tablet_replica.cc:330] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca: stopping tablet replica
I20260806 12:27:48.293363 23922 raft_consensus.cc:2229] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:27:48.293390 23924 raft_consensus.cc:2258] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:48.293710 23922 raft_consensus.cc:2258] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:48.292932 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 23204
I20260806 12:27:48.295271 23924 ts_tablet_manager.cc:1905] T c31ffbc8bd434772914164d925d056f7 P 1bd90b4e8a8e44269f4ad12d8fb0c1d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:48.295924 23921 ts_tablet_manager.cc:1905] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:48.298933 23921 ts_tablet_manager.cc:1918] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:48.299088 23921 log.cc:1196] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/c31ffbc8bd434772914164d925d056f7
I20260806 12:27:48.299451 23921 ts_tablet_manager.cc:1939] T c31ffbc8bd434772914164d925d056f7 P b893095cd36e4fd9910183c998cfffd6: Deleting consensus metadata
I20260806 12:27:48.300719 23922 ts_tablet_manager.cc:1905] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:48.301167 23067 catalog_manager.cc:4841] TS b893095cd36e4fd9910183c998cfffd6 (127.28.210.66:36915): tablet c31ffbc8bd434772914164d925d056f7 (table test-table [id=749cd17593704e2dbb7e9790cf59ddd7]) successfully deleted
I20260806 12:27:48.304293 23922 ts_tablet_manager.cc:1918] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:48.304466 23922 log.cc:1196] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/c31ffbc8bd434772914164d925d056f7
I20260806 12:27:48.304842 23922 ts_tablet_manager.cc:1939] T c31ffbc8bd434772914164d925d056f7 P db94adced85e426c8b020c95ff95deca: Deleting consensus metadata
I20260806 12:27:48.306710 23069 catalog_manager.cc:4841] TS db94adced85e426c8b020c95ff95deca (127.28.210.67:36723): tablet c31ffbc8bd434772914164d925d056f7 (table test-table [id=749cd17593704e2dbb7e9790cf59ddd7]) successfully deleted
I20260806 12:27:48.311004 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 23428
W20260806 12:27:48.310962 23069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:36735 (error 108)
I20260806 12:27:48.315223 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.314775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:36735 (local address 127.0.0.1:37314)
0806 12:27:48.315030 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:48.315111 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:36735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260806 12:27:48.315440 23069 catalog_manager.cc:4584] TS 1bd90b4e8a8e44269f4ad12d8fb0c1d9 (127.28.210.65:36735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c31ffbc8bd434772914164d925d056f7: Network error: Client connection negotiation failed: client connection to 127.28.210.65:36735: connect: Connection refused (error 111)
I20260806 12:27:48.332386 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 23629
I20260806 12:27:48.354773 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 23040
2026-08-06T12:27:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2856 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T12:27:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:27:48Z Disabled control of system clock
I20260806 12:27:48.405068 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:35339
--webserver_interface=127.28.210.126
--webserver_port=0
--builtin_ntp_servers=127.28.210.84:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:35339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 12:27:48.534323 23929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:48.534664 23929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:48.534701 23929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:48.538946 23929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:27:48.539036 23929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:48.539050 23929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:48.539067 23929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:27:48.539080 23929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:27:48.544893 23929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:35339
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:35339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:48.546396 23929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:48.548422 23929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:48.558297 23940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:48.558348 23937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:48.558290 23942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:48.558693 23929 server_base.cc:1034] running on GCE node
I20260806 12:27:48.559424 23929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:48.560708 23929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:48.561887 23929 hybrid_clock.cc:648] HybridClock initialized: now 1786019268561876 us; error 52 us; skew 500 ppm
I20260806 12:27:48.564064 23929 webserver.cc:460] Webserver started at http://127.28.210.126:44031/ using document root <none> and password file <none>
I20260806 12:27:48.564725 23929 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:48.564846 23929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:48.565095 23929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:48.567288 23929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "5a37a846b4814df09a0213c6d7718c7d"
format_stamp: "Formatted at 2026-08-06 12:27:48 on dist-test-slave-lgr5"
server_key: "e7079873eb0fa92f3872b4cd5c53810f"
server_key_iv: "6d160d1dbcb43d4cdc7ea1664a431fda"
server_key_version: "encryptionkey@0"
I20260806 12:27:48.567878 23929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "5a37a846b4814df09a0213c6d7718c7d"
format_stamp: "Formatted at 2026-08-06 12:27:48 on dist-test-slave-lgr5"
server_key: "e7079873eb0fa92f3872b4cd5c53810f"
server_key_iv: "6d160d1dbcb43d4cdc7ea1664a431fda"
server_key_version: "encryptionkey@0"
I20260806 12:27:48.571892 23929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 12:27:48.574666 23955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:48.576391 23929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:48.576560 23929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "5a37a846b4814df09a0213c6d7718c7d"
format_stamp: "Formatted at 2026-08-06 12:27:48 on dist-test-slave-lgr5"
server_key: "e7079873eb0fa92f3872b4cd5c53810f"
server_key_iv: "6d160d1dbcb43d4cdc7ea1664a431fda"
server_key_version: "encryptionkey@0"
I20260806 12:27:48.576694 23929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:48.588562 23929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:48.589377 23929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:48.589570 23929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:48.597497 23929 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:35339
I20260806 12:27:48.597528 24034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:35339 every 8 connection(s)
I20260806 12:27:48.598582 23929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:27:48.601956 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:48.603211 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 23929
I20260806 12:27:48.603358 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:27:48.603686 29513 external_mini_cluster.cc:1428] Setting key cd2db259c125830512589ee77679ab25
I20260806 12:27:48.608943 24035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d: Bootstrap starting.
I20260806 12:27:48.611533 24035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:48.612433 24035 log.cc:824] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:48.614012 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.605474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.28.210.126:35339)
0806 12:27:48.606271 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:48.606287 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:48.606326 (+ 39us) server_negotiation.cc:407] Connection header received
0806 12:27:48.607023 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:48.607045 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:48.607497 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:48.607837 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:48.608379 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.609430 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.610333 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.610847 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.611013 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.611955 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:48.611960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:48.611975 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:48.612478 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:48.612548 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:48.612578 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:48.612677 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:48.612810 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 12:27:48.613095 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":180,"threads_started":1}
I20260806 12:27:48.614732 24035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d: No bootstrap required, opened a new log
I20260806 12:27:48.618695 24035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37a846b4814df09a0213c6d7718c7d" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35339 } }
I20260806 12:27:48.618974 24035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:48.619046 24035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a37a846b4814df09a0213c6d7718c7d, State: Initialized, Role: FOLLOWER
I20260806 12:27:48.619484 24035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37a846b4814df09a0213c6d7718c7d" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35339 } }
I20260806 12:27:48.619650 24035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:27:48.619721 24035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:27:48.619827 24035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:48.621050 24035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37a846b4814df09a0213c6d7718c7d" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35339 } }
I20260806 12:27:48.621413 24035 leader_election.cc:302] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a37a846b4814df09a0213c6d7718c7d; no voters:
I20260806 12:27:48.621749 24035 leader_election.cc:288] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:27:48.621914 24041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:48.622154 24041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [term 1 LEADER]: Becoming Leader. State: Replica: 5a37a846b4814df09a0213c6d7718c7d, State: Running, Role: LEADER
I20260806 12:27:48.622493 24041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37a846b4814df09a0213c6d7718c7d" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35339 } }
I20260806 12:27:48.623162 24035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:27:48.624413 24043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a37a846b4814df09a0213c6d7718c7d. Latest consensus state: current_term: 1 leader_uuid: "5a37a846b4814df09a0213c6d7718c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37a846b4814df09a0213c6d7718c7d" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35339 } } }
I20260806 12:27:48.624385 24042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a37a846b4814df09a0213c6d7718c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37a846b4814df09a0213c6d7718c7d" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35339 } } }
I20260806 12:27:48.624545 24043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:48.624545 24042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:48.624975 24055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:27:48.628211 24055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:27:48.634660 24055 catalog_manager.cc:1347] Generated new cluster ID: 206a0113d4da464885f8cc874ebe393d
I20260806 12:27:48.634758 24055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:27:48.641626 24055 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:27:48.643041 24055 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:27:48.659693 24055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d: Generated new TSK 0
I20260806 12:27:48.661027 24055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:27:48.669437 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35339
--builtin_ntp_servers=127.28.210.84:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 12:27:48.798698 24069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:48.799041 24069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:48.799085 24069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:48.803618 24069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:48.803714 24069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:48.803773 24069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:27:48.809118 24069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:48.810338 24069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:48.811893 24069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:48.824446 24079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:48.824481 24080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:48.824577 24069 server_base.cc:1034] running on GCE node
W20260806 12:27:48.824436 24082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:48.825241 24069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:48.826165 24069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:48.827423 24069 hybrid_clock.cc:648] HybridClock initialized: now 1786019268827398 us; error 41 us; skew 500 ppm
I20260806 12:27:48.829969 24069 webserver.cc:460] Webserver started at http://127.28.210.65:34547/ using document root <none> and password file <none>
I20260806 12:27:48.830667 24069 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:48.830749 24069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:48.830971 24069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:48.833762 24069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "3a3cec1d84444f9e9d8a658d5c9616a1"
format_stamp: "Formatted at 2026-08-06 12:27:48 on dist-test-slave-lgr5"
server_key: "ce544377df92ac8f47b2530076a2835c"
server_key_iv: "a458442ebd0f40f0593af4b52c2124db"
server_key_version: "encryptionkey@0"
I20260806 12:27:48.834800 24069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "3a3cec1d84444f9e9d8a658d5c9616a1"
format_stamp: "Formatted at 2026-08-06 12:27:48 on dist-test-slave-lgr5"
server_key: "ce544377df92ac8f47b2530076a2835c"
server_key_iv: "a458442ebd0f40f0593af4b52c2124db"
server_key_version: "encryptionkey@0"
I20260806 12:27:48.839974 24069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 12:27:48.843735 24094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:48.845408 24069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:48.845568 24069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "3a3cec1d84444f9e9d8a658d5c9616a1"
format_stamp: "Formatted at 2026-08-06 12:27:48 on dist-test-slave-lgr5"
server_key: "ce544377df92ac8f47b2530076a2835c"
server_key_iv: "a458442ebd0f40f0593af4b52c2124db"
server_key_version: "encryptionkey@0"
I20260806 12:27:48.845707 24069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:48.861430 24069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:48.862528 24069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:48.862789 24069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:48.863569 24069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:48.865049 24069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:48.865113 24069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:48.865165 24069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:48.865187 24069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:48.876040 24069 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:36669
I20260806 12:27:48.876205 24267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:36669 every 8 connection(s)
I20260806 12:27:48.877265 24069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:27:48.878337 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 24069
I20260806 12:27:48.878480 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:27:48.878860 29513 external_mini_cluster.cc:1428] Setting key e47e695df5b886a56d98792a5c88a976
I20260806 12:27:48.881803 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35339
--builtin_ntp_servers=127.28.210.84:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 12:27:48.893540 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.881590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:55497 (local address 127.28.210.126:35339)
0806 12:27:48.881741 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:48.881746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:48.883765 (+ 2019us) server_negotiation.cc:407] Connection header received
0806 12:27:48.885925 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:48.885930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:48.886018 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:48.886118 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:48.887765 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.888500 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.890719 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.890985 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.891125 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.892850 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:48.892855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:48.892857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:48.892876 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:48.892901 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:48.892911 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:48.892980 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:48.893291 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 12:27:48.893386 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 12:27:48.895383 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:48.882311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35339 (local address 127.28.210.65:55497)
0806 12:27:48.883392 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:48.883472 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:48.885514 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:48.886420 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:48.886432 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:48.886594 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:48.887483 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.887510 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.888700 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.888704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.890505 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:48.890528 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:48.891212 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:48.891218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:48.891490 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:48.892385 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:27:48.892387 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:48.892465 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:27:48.892596 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:48.893016 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:48.893031 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:48.893038 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 12:27:48.893229 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 12:27:48.893804 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":173,"threads_started":1}
I20260806 12:27:48.897223 24269 heartbeater.cc:347] Connected to a master server at 127.28.210.126:35339
I20260806 12:27:48.897562 24269 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:48.898252 24269 heartbeater.cc:511] Master 127.28.210.126:35339 requested a full tablet report, sending...
I20260806 12:27:48.900383 23986 ts_manager.cc:194] Registered new tserver with Master: 3a3cec1d84444f9e9d8a658d5c9616a1 (127.28.210.65:36669)
I20260806 12:27:48.901782 23986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.210.65:55497
W20260806 12:27:49.018599 24275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:49.018949 24275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:49.018992 24275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:49.023895 24275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:49.023995 24275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:49.024115 24275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:27:49.029670 24275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:49.031028 24275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:49.032629 24275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:49.044572 24294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:49.044591 24286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:49.044572 24287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:49.044878 24275 server_base.cc:1034] running on GCE node
I20260806 12:27:49.045437 24275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:49.046240 24275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:49.047451 24275 hybrid_clock.cc:648] HybridClock initialized: now 1786019269047400 us; error 86 us; skew 500 ppm
I20260806 12:27:49.049902 24275 webserver.cc:460] Webserver started at http://127.28.210.66:39723/ using document root <none> and password file <none>
I20260806 12:27:49.050591 24275 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:49.050686 24275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:49.050938 24275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:49.053617 24275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "65c30a3a59394397b56f981ec65c6935"
format_stamp: "Formatted at 2026-08-06 12:27:49 on dist-test-slave-lgr5"
server_key: "dd7ec09bfbdef605aaa3b5f446a597f3"
server_key_iv: "368e5bf739fbb7cd4eebbd2747c6ede1"
server_key_version: "encryptionkey@0"
I20260806 12:27:49.054304 24275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "65c30a3a59394397b56f981ec65c6935"
format_stamp: "Formatted at 2026-08-06 12:27:49 on dist-test-slave-lgr5"
server_key: "dd7ec09bfbdef605aaa3b5f446a597f3"
server_key_iv: "368e5bf739fbb7cd4eebbd2747c6ede1"
server_key_version: "encryptionkey@0"
I20260806 12:27:49.059250 24275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 12:27:49.062994 24300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:49.064745 24275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:49.064927 24275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "65c30a3a59394397b56f981ec65c6935"
format_stamp: "Formatted at 2026-08-06 12:27:49 on dist-test-slave-lgr5"
server_key: "dd7ec09bfbdef605aaa3b5f446a597f3"
server_key_iv: "368e5bf739fbb7cd4eebbd2747c6ede1"
server_key_version: "encryptionkey@0"
I20260806 12:27:49.065073 24275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:49.086920 24275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:49.087723 24275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:49.087925 24275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:49.088637 24275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:49.089700 24275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:49.089752 24275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:49.089818 24275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:49.089844 24275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:49.099105 24275 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:42017
I20260806 12:27:49.099138 24462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:42017 every 8 connection(s)
I20260806 12:27:49.100136 24275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:27:49.102540 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 24275
I20260806 12:27:49.102684 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:27:49.103051 29513 external_mini_cluster.cc:1428] Setting key f754eab1d1f4dc2f80899fde6c8fbdd9
I20260806 12:27:49.105350 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35339
--builtin_ntp_servers=127.28.210.84:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 12:27:49.111840 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.102250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:37563 (local address 127.28.210.126:35339)
0806 12:27:49.102422 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:49.102426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:49.104116 (+ 1690us) server_negotiation.cc:407] Connection header received
0806 12:27:49.105881 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:49.105884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:49.105937 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:49.106014 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:49.107277 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.108111 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.109359 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.109665 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.109783 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.111228 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:49.111233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:49.111236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:49.111257 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:49.111285 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:49.111297 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:49.111387 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:49.111586 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 12:27:49.111666 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:27:49.112790 24466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.102814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35339 (local address 127.28.210.66:37563)
0806 12:27:49.103970 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:49.104015 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:49.105605 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:49.106233 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:49.106243 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:49.106359 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:49.107074 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.107095 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.108248 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.108251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.109218 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.109230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.109921 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.109925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.110124 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.110905 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:27:49.110907 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:49.110958 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:27:49.111042 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:49.111418 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:49.111428 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:49.111434 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 12:27:49.111567 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 12:27:49.111843 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":174,"threads_started":1}
I20260806 12:27:49.114439 24464 heartbeater.cc:347] Connected to a master server at 127.28.210.126:35339
I20260806 12:27:49.114763 24464 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:49.115597 24464 heartbeater.cc:511] Master 127.28.210.126:35339 requested a full tablet report, sending...
I20260806 12:27:49.117244 23986 ts_manager.cc:194] Registered new tserver with Master: 65c30a3a59394397b56f981ec65c6935 (127.28.210.66:42017)
I20260806 12:27:49.117928 23986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.210.66:37563
W20260806 12:27:49.235651 24467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:49.236003 24467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:49.236065 24467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:49.241017 24467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:49.241127 24467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:49.241225 24467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:27:49.247040 24467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:35339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:49.248528 24467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:49.250155 24467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:49.261061 24476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:49.261089 24477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:49.261161 24467 server_base.cc:1034] running on GCE node
W20260806 12:27:49.261099 24481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:49.261901 24467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:49.262873 24467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:49.264127 24467 hybrid_clock.cc:648] HybridClock initialized: now 1786019269264055 us; error 47 us; skew 500 ppm
I20260806 12:27:49.266508 24467 webserver.cc:460] Webserver started at http://127.28.210.67:40515/ using document root <none> and password file <none>
I20260806 12:27:49.267236 24467 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:49.267311 24467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:49.267544 24467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:49.270130 24467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb"
format_stamp: "Formatted at 2026-08-06 12:27:49 on dist-test-slave-lgr5"
server_key: "6af87e3417a7ba75827d2d7917a7834a"
server_key_iv: "49fede6551b0997a05eda070698c45fb"
server_key_version: "encryptionkey@0"
I20260806 12:27:49.270846 24467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb"
format_stamp: "Formatted at 2026-08-06 12:27:49 on dist-test-slave-lgr5"
server_key: "6af87e3417a7ba75827d2d7917a7834a"
server_key_iv: "49fede6551b0997a05eda070698c45fb"
server_key_version: "encryptionkey@0"
I20260806 12:27:49.275581 24467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 12:27:49.279189 24490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:49.280926 24467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:49.281086 24467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb"
format_stamp: "Formatted at 2026-08-06 12:27:49 on dist-test-slave-lgr5"
server_key: "6af87e3417a7ba75827d2d7917a7834a"
server_key_iv: "49fede6551b0997a05eda070698c45fb"
server_key_version: "encryptionkey@0"
I20260806 12:27:49.281240 24467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:49.301846 24467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:49.302863 24467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:49.303131 24467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:49.303956 24467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:49.305191 24467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:49.305258 24467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:49.305307 24467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:49.305321 24467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:49.316058 24467 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:45905
I20260806 12:27:49.316134 24655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:45905 every 8 connection(s)
I20260806 12:27:49.317299 24467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:27:49.324798 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 24467
I20260806 12:27:49.324934 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:27:49.325222 29513 external_mini_cluster.cc:1428] Setting key 40d2541e3d8d905fa85707533d8da960
I20260806 12:27:49.329185 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.319824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:55243 (local address 127.28.210.126:35339)
0806 12:27:49.320289 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:49.320294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:49.321418 (+ 1124us) server_negotiation.cc:407] Connection header received
0806 12:27:49.323149 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:49.323153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:49.323227 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:49.323310 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:49.324614 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.325321 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.326748 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.327058 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.327208 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.328690 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:49.328694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:49.328696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:49.328711 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:49.328736 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:49.328744 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:49.328810 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:49.328972 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 12:27:49.329042 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 12:27:49.330302 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.320322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:35339 (local address 127.28.210.67:55243)
0806 12:27:49.321213 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:49.321266 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:49.322860 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:49.323556 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:49.323592 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:49.323716 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:49.324411 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.324429 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.325509 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.325513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.326587 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.326601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.327412 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.327416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.327612 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.328363 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:27:49.328365 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:49.328417 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:27:49.328511 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:49.328822 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:49.328831 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:49.328836 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 12:27:49.328947 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 12:27:49.329275 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":157,"threads_started":1}
I20260806 12:27:49.331722 24657 heartbeater.cc:347] Connected to a master server at 127.28.210.126:35339
I20260806 12:27:49.332047 24657 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:49.332800 24657 heartbeater.cc:511] Master 127.28.210.126:35339 requested a full tablet report, sending...
I20260806 12:27:49.334198 23986 ts_manager.cc:194] Registered new tserver with Master: 030e77d5e8074f9cb6cbe0e7e6dbc4fb (127.28.210.67:45905)
I20260806 12:27:49.334862 23986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.210.67:55243
I20260806 12:27:49.341149 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:27:49.348810 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.343581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.28.210.126:35339)
0806 12:27:49.343770 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:49.343775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:49.343887 (+ 112us) server_negotiation.cc:407] Connection header received
0806 12:27:49.344056 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:49.344059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:49.344136 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:49.344211 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:49.344662 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.345261 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.346544 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.346798 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.346912 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.348188 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:49.348193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:49.348195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:49.348220 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:49.348252 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:49.348262 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:49.348340 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:49.348457 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 12:27:49.348589 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 12:27:49.354221 23986 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:27:49.357982 23986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:27:49.378371 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.370940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.28.210.65:36669)
0806 12:27:49.372542 (+ 1602us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:49.372551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:49.372579 (+ 28us) server_negotiation.cc:407] Connection header received
0806 12:27:49.372678 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:49.372682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:49.373005 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:49.373228 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:49.373777 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.375127 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.376672 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.377563 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.377681 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.377928 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:49.378064 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 12:27:49.378187 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1468,"thread_start_us":105,"threads_started":1}
I20260806 12:27:49.378371 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.370768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:36669 (local address 127.0.0.1:47186)
0806 12:27:49.371980 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:49.371995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:49.372182 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:49.373352 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:49.373362 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:49.373399 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:49.373610 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.373617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.375263 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.375267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.376506 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.376520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.377697 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.377700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.377835 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.377876 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 12:27:49.378018 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 12:27:49.378187 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":26,"spinlock_wait_cycles":6912,"thread_start_us":143,"threads_started":1}
I20260806 12:27:49.378590 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.370539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:45905 (local address 127.0.0.1:43334)
0806 12:27:49.371904 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:49.371959 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:49.372183 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:49.373933 (+ 1750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:49.373936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:49.373958 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:49.374080 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.374087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.375615 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.375619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.376970 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.376983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.378128 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.378132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.378225 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.378248 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 12:27:49.378301 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 12:27:49.378380 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"mutex_wait_us":110,"spinlock_wait_cycles":10368,"thread_start_us":249,"threads_started":1}
I20260806 12:27:49.379391 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.372312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.28.210.66:42017)
0806 12:27:49.373907 (+ 1595us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:49.373921 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:49.373964 (+ 43us) server_negotiation.cc:407] Connection header received
0806 12:27:49.374054 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:49.374058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:49.374218 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:49.374359 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:49.374617 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.375518 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.377132 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.377976 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.378094 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.378315 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:49.379094 (+ 779us) server_negotiation.cc:299] Negotiation successful
0806 12:27:49.379223 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1468,"thread_start_us":118,"threads_started":1}
I20260806 12:27:49.379261 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.372112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:42017 (local address 127.0.0.1:50220)
0806 12:27:49.372586 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:49.372602 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:49.372730 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:49.374361 (+ 1631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:49.374365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:49.374383 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:49.374499 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.374507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.375641 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.375644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.376974 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.376983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.378377 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.378381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.378944 (+ 563us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.378969 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 12:27:49.379048 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 12:27:49.379125 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":27,"thread_start_us":119,"threads_started":1}
I20260806 12:27:49.380273 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.370956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.28.210.67:45905)
0806 12:27:49.373234 (+ 2278us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:49.373376 (+ 142us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:49.373424 (+ 48us) server_negotiation.cc:407] Connection header received
0806 12:27:49.373522 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:49.373528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:49.373754 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:49.374258 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:49.374438 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.375497 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.377132 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.377976 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.378929 (+ 953us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.379144 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:49.379201 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 12:27:49.379475 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2036,"thread_start_us":171,"threads_started":1}
I20260806 12:27:49.382615 24169 tablet_service.cc:1467] Processing CreateTablet for tablet 7b994d42f89c4ff6888c8a717b8bf135 (DEFAULT_TABLE table=test-table [id=16f6c38ef56d4bb39d2f0daeb4ce9db0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:49.382982 24365 tablet_service.cc:1467] Processing CreateTablet for tablet 7b994d42f89c4ff6888c8a717b8bf135 (DEFAULT_TABLE table=test-table [id=16f6c38ef56d4bb39d2f0daeb4ce9db0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:49.383222 24561 tablet_service.cc:1467] Processing CreateTablet for tablet 7b994d42f89c4ff6888c8a717b8bf135 (DEFAULT_TABLE table=test-table [id=16f6c38ef56d4bb39d2f0daeb4ce9db0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:49.384038 24365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b994d42f89c4ff6888c8a717b8bf135. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:49.384037 24169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b994d42f89c4ff6888c8a717b8bf135. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:49.384274 24561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b994d42f89c4ff6888c8a717b8bf135. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:49.393344 24688 tablet_bootstrap.cc:492] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935: Bootstrap starting.
I20260806 12:27:49.393734 24692 tablet_bootstrap.cc:492] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1: Bootstrap starting.
I20260806 12:27:49.395916 24692 tablet_bootstrap.cc:654] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:49.395926 24688 tablet_bootstrap.cc:654] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:49.396649 24689 tablet_bootstrap.cc:492] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb: Bootstrap starting.
I20260806 12:27:49.396979 24692 log.cc:824] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:49.396999 24688 log.cc:824] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:49.399370 24689 tablet_bootstrap.cc:654] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:49.399788 24688 tablet_bootstrap.cc:492] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935: No bootstrap required, opened a new log
I20260806 12:27:49.400033 24688 ts_tablet_manager.cc:1397] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260806 12:27:49.400439 24689 log.cc:824] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:49.401312 24692 tablet_bootstrap.cc:492] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1: No bootstrap required, opened a new log
I20260806 12:27:49.401537 24692 ts_tablet_manager.cc:1397] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260806 12:27:49.403959 24688 raft_consensus.cc:348] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45905 } } peers { permanent_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36669 } } peers { permanent_uuid: "65c30a3a59394397b56f981ec65c6935" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42017 } }
I20260806 12:27:49.404301 24688 raft_consensus.cc:374] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:49.404387 24688 raft_consensus.cc:729] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c30a3a59394397b56f981ec65c6935, State: Initialized, Role: FOLLOWER
I20260806 12:27:49.404920 24688 consensus_queue.cc:260] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45905 } } peers { permanent_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36669 } } peers { permanent_uuid: "65c30a3a59394397b56f981ec65c6935" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42017 } }
I20260806 12:27:49.406319 24464 heartbeater.cc:503] Master 127.28.210.126:35339 was elected leader, sending a full tablet report...
I20260806 12:27:49.406386 24688 ts_tablet_manager.cc:1428] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 12:27:49.406920 24689 tablet_bootstrap.cc:492] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb: No bootstrap required, opened a new log
I20260806 12:27:49.406621 24692 raft_consensus.cc:348] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45905 } } peers { permanent_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36669 } } peers { permanent_uuid: "65c30a3a59394397b56f981ec65c6935" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42017 } }
I20260806 12:27:49.407114 24692 raft_consensus.cc:374] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:49.407192 24692 raft_consensus.cc:729] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3cec1d84444f9e9d8a658d5c9616a1, State: Initialized, Role: FOLLOWER
I20260806 12:27:49.407188 24689 ts_tablet_manager.cc:1397] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260806 12:27:49.407740 24692 consensus_queue.cc:260] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45905 } } peers { permanent_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36669 } } peers { permanent_uuid: "65c30a3a59394397b56f981ec65c6935" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42017 } }
I20260806 12:27:49.409451 24269 heartbeater.cc:503] Master 127.28.210.126:35339 was elected leader, sending a full tablet report...
I20260806 12:27:49.409979 24692 ts_tablet_manager.cc:1428] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1: Time spent starting tablet: real 0.008s user 0.006s sys 0.003s
I20260806 12:27:49.411914 24689 raft_consensus.cc:348] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45905 } } peers { permanent_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36669 } } peers { permanent_uuid: "65c30a3a59394397b56f981ec65c6935" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42017 } }
I20260806 12:27:49.412274 24689 raft_consensus.cc:374] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:49.412350 24689 raft_consensus.cc:729] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 030e77d5e8074f9cb6cbe0e7e6dbc4fb, State: Initialized, Role: FOLLOWER
I20260806 12:27:49.412873 24689 consensus_queue.cc:260] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45905 } } peers { permanent_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36669 } } peers { permanent_uuid: "65c30a3a59394397b56f981ec65c6935" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42017 } }
I20260806 12:27:49.413795 24657 heartbeater.cc:503] Master 127.28.210.126:35339 was elected leader, sending a full tablet report...
I20260806 12:27:49.414156 24689 ts_tablet_manager.cc:1428] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260806 12:27:49.568457 24661 tablet.cc:2369] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:27:49.601651 24465 tablet.cc:2369] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:27:49.629026 24271 tablet.cc:2369] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:27:49.771077 24700 raft_consensus.cc:482] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:27:49.771363 24700 raft_consensus.cc:504] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45905 } } peers { permanent_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36669 } } peers { permanent_uuid: "65c30a3a59394397b56f981ec65c6935" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42017 } }
I20260806 12:27:49.772918 24700 leader_election.cc:288] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 030e77d5e8074f9cb6cbe0e7e6dbc4fb (127.28.210.67:45905), 3a3cec1d84444f9e9d8a658d5c9616a1 (127.28.210.65:36669)
I20260806 12:27:49.777451 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.773301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:57603 (local address 127.28.210.65:36669)
0806 12:27:49.773465 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:49.773469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:49.773762 (+ 293us) server_negotiation.cc:407] Connection header received
0806 12:27:49.773899 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:49.773903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:49.773959 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:49.774024 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:49.774354 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.774930 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.776287 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.776734 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.776873 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.776944 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:49.777210 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 12:27:49.777300 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 12:27:49.777474 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.773208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:45905 (local address 127.28.210.66:49339)
0806 12:27:49.773718 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:49.773736 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:49.773883 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:49.774169 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:49.774171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:49.774187 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:49.774295 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.774301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.775090 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.775094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.776119 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.776138 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.776891 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.776895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.777071 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.777092 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 12:27:49.777169 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 12:27:49.777286 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":96,"thread_start_us":108,"threads_started":1}
I20260806 12:27:49.777451 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.773301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:49339 (local address 127.28.210.67:45905)
0806 12:27:49.773461 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:49.773466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:49.773832 (+ 366us) server_negotiation.cc:407] Connection header received
0806 12:27:49.773996 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:49.773999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:49.774054 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:49.774137 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:49.774414 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.774944 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.776295 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.776747 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.776880 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.776958 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:49.777208 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 12:27:49.777296 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:27:49.777482 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.773208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:36669 (local address 127.28.210.66:57603)
0806 12:27:49.773641 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:49.773655 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:49.773783 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:49.774058 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:49.774061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:49.774078 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:49.774233 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.774241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.775091 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.775094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.776134 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:49.776149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.776885 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.776888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:49.777061 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.777087 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 12:27:49.777164 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 12:27:49.777294 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":129,"thread_start_us":155,"threads_started":1}
I20260806 12:27:49.778342 24588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b994d42f89c4ff6888c8a717b8bf135" candidate_uuid: "65c30a3a59394397b56f981ec65c6935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb" is_pre_election: true
I20260806 12:27:49.778321 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b994d42f89c4ff6888c8a717b8bf135" candidate_uuid: "65c30a3a59394397b56f981ec65c6935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1" is_pre_election: true
I20260806 12:27:49.778762 24205 raft_consensus.cc:2454] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c30a3a59394397b56f981ec65c6935 in term 0.
I20260806 12:27:49.778761 24588 raft_consensus.cc:2454] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c30a3a59394397b56f981ec65c6935 in term 0.
I20260806 12:27:49.779408 24301 leader_election.cc:302] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3cec1d84444f9e9d8a658d5c9616a1, 65c30a3a59394397b56f981ec65c6935; no voters:
I20260806 12:27:49.779773 24700 raft_consensus.cc:2781] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:27:49.779887 24700 raft_consensus.cc:482] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:27:49.779945 24700 raft_consensus.cc:3037] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:49.781273 24700 raft_consensus.cc:504] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45905 } } peers { permanent_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36669 } } peers { permanent_uuid: "65c30a3a59394397b56f981ec65c6935" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42017 } }
I20260806 12:27:49.781754 24700 leader_election.cc:288] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [CANDIDATE]: Term 1 election: Requested vote from peers 030e77d5e8074f9cb6cbe0e7e6dbc4fb (127.28.210.67:45905), 3a3cec1d84444f9e9d8a658d5c9616a1 (127.28.210.65:36669)
I20260806 12:27:49.782368 24588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b994d42f89c4ff6888c8a717b8bf135" candidate_uuid: "65c30a3a59394397b56f981ec65c6935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb"
I20260806 12:27:49.782469 24205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b994d42f89c4ff6888c8a717b8bf135" candidate_uuid: "65c30a3a59394397b56f981ec65c6935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1"
I20260806 12:27:49.782539 24588 raft_consensus.cc:3037] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:49.782711 24205 raft_consensus.cc:3037] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:49.783869 24588 raft_consensus.cc:2454] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c30a3a59394397b56f981ec65c6935 in term 1.
I20260806 12:27:49.783994 24205 raft_consensus.cc:2454] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c30a3a59394397b56f981ec65c6935 in term 1.
I20260806 12:27:49.784422 24302 leader_election.cc:302] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 030e77d5e8074f9cb6cbe0e7e6dbc4fb, 65c30a3a59394397b56f981ec65c6935; no voters:
I20260806 12:27:49.784648 24700 raft_consensus.cc:2781] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:49.784942 24700 raft_consensus.cc:686] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 1 LEADER]: Becoming Leader. State: Replica: 65c30a3a59394397b56f981ec65c6935, State: Running, Role: LEADER
I20260806 12:27:49.785383 24700 consensus_queue.cc:237] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45905 } } peers { permanent_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36669 } } peers { permanent_uuid: "65c30a3a59394397b56f981ec65c6935" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42017 } }
I20260806 12:27:49.789191 23986 catalog_manager.cc:5495] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c30a3a59394397b56f981ec65c6935 (127.28.210.66). New cstate: current_term: 1 leader_uuid: "65c30a3a59394397b56f981ec65c6935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c30a3a59394397b56f981ec65c6935" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42017 } health_report { overall_health: HEALTHY } } }
I20260806 12:27:49.837169 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.832787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.28.210.66:42017)
0806 12:27:49.832917 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:49.832921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:49.832937 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:27:49.833185 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:49.833227 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:49.833420 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:49.833606 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:27:49.834009 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.834601 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.835590 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:49.835822 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:49.835912 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:49.836231 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:27:49.836762 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:27:49.836863 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:49.836971 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 12:27:49.837034 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 12:27:49.844412 24205 raft_consensus.cc:1261] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1 [term 1 FOLLOWER]: Refusing update from remote peer 65c30a3a59394397b56f981ec65c6935: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:49.844437 24588 raft_consensus.cc:1261] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb [term 1 FOLLOWER]: Refusing update from remote peer 65c30a3a59394397b56f981ec65c6935: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:49.845291 24700 consensus_queue.cc:1035] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "030e77d5e8074f9cb6cbe0e7e6dbc4fb" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:49.845546 24708 consensus_queue.cc:1035] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3cec1d84444f9e9d8a658d5c9616a1" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 36669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:49.856343 24715 mvcc.cc:205] Tried to move back new op lower bound from 7315534929273565184 to 7315534929044185088. Current Snapshot: MvccSnapshot[applied={T|T < 7315534929273565184}]
I20260806 12:27:49.868479 23986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:49.868714 23986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:49.871752 23986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a37a846b4814df09a0213c6d7718c7d: Sending DeleteTablet for 3 replicas of tablet 7b994d42f89c4ff6888c8a717b8bf135
I20260806 12:27:49.872872 24169 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b994d42f89c4ff6888c8a717b8bf135 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:49 UTC) from {username='slave'} at 127.0.0.1:47186
I20260806 12:27:49.872915 24561 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b994d42f89c4ff6888c8a717b8bf135 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:49 UTC) from {username='slave'} at 127.0.0.1:43334
I20260806 12:27:49.873116 24365 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b994d42f89c4ff6888c8a717b8bf135 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:49 UTC) from {username='slave'} at 127.0.0.1:50220
I20260806 12:27:49.873613 24731 tablet_replica.cc:330] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935: stopping tablet replica
I20260806 12:27:49.873802 24730 tablet_replica.cc:330] T 7b994d42f89c4ff6888c8a717b8bf135 P 3a3cec1d84444f9e9d8a658d5c9616a1: stopping tablet replica
I20260806 12:27:49.873839 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 24069
I20260806 12:27:49.874055 24731 raft_consensus.cc:2229] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:27:49.874522 24729 tablet_replica.cc:330] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb: stopping tablet replica
I20260806 12:27:49.874778 24731 raft_consensus.cc:2258] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:49.877460 24729 raft_consensus.cc:2229] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:27:49.877960 24729 raft_consensus.cc:2258] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:49.884738 24731 ts_tablet_manager.cc:1905] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:49.885130 24729 ts_tablet_manager.cc:1905] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:49.888484 24729 ts_tablet_manager.cc:1918] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:49.888577 24731 ts_tablet_manager.cc:1918] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:49.888739 24731 log.cc:1196] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/7b994d42f89c4ff6888c8a717b8bf135
I20260806 12:27:49.888859 24729 log.cc:1196] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/7b994d42f89c4ff6888c8a717b8bf135
I20260806 12:27:49.889168 24731 ts_tablet_manager.cc:1939] T 7b994d42f89c4ff6888c8a717b8bf135 P 65c30a3a59394397b56f981ec65c6935: Deleting consensus metadata
I20260806 12:27:49.889405 24729 ts_tablet_manager.cc:1939] T 7b994d42f89c4ff6888c8a717b8bf135 P 030e77d5e8074f9cb6cbe0e7e6dbc4fb: Deleting consensus metadata
I20260806 12:27:49.890974 23957 catalog_manager.cc:4841] TS 030e77d5e8074f9cb6cbe0e7e6dbc4fb (127.28.210.67:45905): tablet 7b994d42f89c4ff6888c8a717b8bf135 (table test-table [id=16f6c38ef56d4bb39d2f0daeb4ce9db0]) successfully deleted
I20260806 12:27:49.892843 23957 catalog_manager.cc:4841] TS 65c30a3a59394397b56f981ec65c6935 (127.28.210.66:42017): tablet 7b994d42f89c4ff6888c8a717b8bf135 (table test-table [id=16f6c38ef56d4bb39d2f0daeb4ce9db0]) successfully deleted
W20260806 12:27:49.897151 23956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:36669 (error 108)
I20260806 12:27:49.897943 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 24275
I20260806 12:27:49.906309 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:49.898167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:36669 (local address 127.0.0.1:53664)
0806 12:27:49.905963 (+ 7796us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:49.906150 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:36669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7640,"thread_start_us":77,"threads_started":1}
W20260806 12:27:49.906760 23956 catalog_manager.cc:4584] TS 3a3cec1d84444f9e9d8a658d5c9616a1 (127.28.210.65:36669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b994d42f89c4ff6888c8a717b8bf135: Network error: Client connection negotiation failed: client connection to 127.28.210.65:36669: connect: Connection refused (error 111)
I20260806 12:27:49.913708 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 24467
I20260806 12:27:49.936198 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 23929
2026-08-06T12:27:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1576 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:27:50 dist-test-slave-lgr5 krb5kdc[24740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:27:50 dist-test-slave-lgr5 krb5kdc[24740](info): set up 2 sockets
Aug 06 12:27:50 dist-test-slave-lgr5 krb5kdc[24740](info): commencing operation
krb5kdc: starting...
W20260806 12:27:51.159931 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:27:51 dist-test-slave-lgr5 krb5kdc[24740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:27:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:27:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:27:51.309257 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:46735
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:46735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:51.442965 24755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:51.443344 24755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:51.443399 24755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:51.448326 24755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:27:51.448422 24755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:51.448442 24755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:51.448459 24755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:27:51.448474 24755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:27:51.454581 24755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:46735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:46735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:51.455910 24755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:51.457080 24755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:51.466326 24767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:51.466331 24762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:51.466328 24763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:51.466496 24755 server_base.cc:1034] running on GCE node
I20260806 12:27:51.467303 24755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:51.468571 24755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:51.469745 24755 hybrid_clock.cc:648] HybridClock initialized: now 1786019271469709 us; error 50 us; skew 500 ppm
Aug 06 12:27:51 dist-test-slave-lgr5 krb5kdc[24740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019271, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:51.478663 24755 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:27:51.479810 24755 webserver.cc:460] Webserver started at http://127.28.210.126:35513/ using document root <none> and password file <none>
I20260806 12:27:51.480458 24755 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:51.480525 24755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:51.480749 24755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:51.482964 24755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "c95100b65d6149cab19d5c2a6178eeca"
format_stamp: "Formatted at 2026-08-06 12:27:51 on dist-test-slave-lgr5"
server_key: "a54fa2e60b897879196fef53074fd328"
server_key_iv: "e3e97dba7523b08ec0d140af857974c6"
server_key_version: "encryptionkey@0"
I20260806 12:27:51.483562 24755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "c95100b65d6149cab19d5c2a6178eeca"
format_stamp: "Formatted at 2026-08-06 12:27:51 on dist-test-slave-lgr5"
server_key: "a54fa2e60b897879196fef53074fd328"
server_key_iv: "e3e97dba7523b08ec0d140af857974c6"
server_key_version: "encryptionkey@0"
I20260806 12:27:51.487908 24755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 12:27:51.490777 24779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:51.492223 24755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:51.492369 24755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "c95100b65d6149cab19d5c2a6178eeca"
format_stamp: "Formatted at 2026-08-06 12:27:51 on dist-test-slave-lgr5"
server_key: "a54fa2e60b897879196fef53074fd328"
server_key_iv: "e3e97dba7523b08ec0d140af857974c6"
server_key_version: "encryptionkey@0"
I20260806 12:27:51.492493 24755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:51.516041 24755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:51.519991 24755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:51.520256 24755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:51.529024 24755 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:46735
I20260806 12:27:51.529110 24860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:46735 every 8 connection(s)
I20260806 12:27:51.530224 24755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:27:51.533797 24864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:51.539096 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 24755
I20260806 12:27:51.539270 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:27:51.539608 29513 external_mini_cluster.cc:1428] Setting key 8f6588cc21a352533345c5792d65f902
I20260806 12:27:51.541744 24864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca: Bootstrap starting.
I20260806 12:27:51.544656 24864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:51.545710 24864 log.cc:824] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:51.547961 24864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca: No bootstrap required, opened a new log
Aug 06 12:27:51 dist-test-slave-lgr5 krb5kdc[24740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:27:51.552054 24864 raft_consensus.cc:348] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95100b65d6149cab19d5c2a6178eeca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46735 } }
I20260806 12:27:51.552368 24864 raft_consensus.cc:374] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:51.552433 24864 raft_consensus.cc:729] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95100b65d6149cab19d5c2a6178eeca, State: Initialized, Role: FOLLOWER
I20260806 12:27:51.552945 24864 consensus_queue.cc:260] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95100b65d6149cab19d5c2a6178eeca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46735 } }
I20260806 12:27:51.553095 24864 raft_consensus.cc:388] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:27:51.553164 24864 raft_consensus.cc:482] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:27:51.553269 24864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:51.554332 24864 raft_consensus.cc:504] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95100b65d6149cab19d5c2a6178eeca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46735 } }
I20260806 12:27:51.554700 24864 leader_election.cc:302] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c95100b65d6149cab19d5c2a6178eeca; no voters:
I20260806 12:27:51.555219 24864 leader_election.cc:288] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:27:51.555352 24869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:51.555554 24869 raft_consensus.cc:686] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [term 1 LEADER]: Becoming Leader. State: Replica: c95100b65d6149cab19d5c2a6178eeca, State: Running, Role: LEADER
I20260806 12:27:51.555914 24869 consensus_queue.cc:237] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95100b65d6149cab19d5c2a6178eeca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46735 } }
I20260806 12:27:51.556674 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:51.542012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.28.210.126:46735)
0806 12:27:51.542537 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:51.542556 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:51.542601 (+ 45us) server_negotiation.cc:407] Connection header received
0806 12:27:51.543405 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:51.543425 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:51.543825 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:51.544238 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:51.545302 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:51.546484 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:51.547424 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:51.547945 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:51.548160 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:51.550672 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:51.550678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:51.550695 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:51.550728 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:51.552792 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:51.553265 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:51.553272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:51.553277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:51.553444 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:51.553724 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:51.553728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:51.553730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:51.554216 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:51.554515 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:51.554813 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 12:27:51.555237 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":166,"threads_started":1}
I20260806 12:27:51.556830 24864 sys_catalog.cc:564] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:27:51.558015 24870 sys_catalog.cc:455] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c95100b65d6149cab19d5c2a6178eeca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95100b65d6149cab19d5c2a6178eeca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46735 } } }
I20260806 12:27:51.558173 24870 sys_catalog.cc:458] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:51.558421 24870 sys_catalog.cc:455] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [sys.catalog]: SysCatalogTable state changed. Reason: New leader c95100b65d6149cab19d5c2a6178eeca. Latest consensus state: current_term: 1 leader_uuid: "c95100b65d6149cab19d5c2a6178eeca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95100b65d6149cab19d5c2a6178eeca" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 46735 } } }
I20260806 12:27:51.558512 24870 sys_catalog.cc:458] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:51.559343 24884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:27:51.562814 24884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:27:51.568815 24884 catalog_manager.cc:1347] Generated new cluster ID: 39edcb17c8cd4b4c993ff0161a3d078d
I20260806 12:27:51.568910 24884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:27:51.577175 24884 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:27:51.578291 24884 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:27:51.590746 24884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca: Generated new TSK 0
I20260806 12:27:51.592135 24884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:27:51.634464 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:46735
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:51.766930 24898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:51.767267 24898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:51.767316 24898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:51.771936 24898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:51.772061 24898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:51.772182 24898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:27:51.778283 24898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:46735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:51.779755 24898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:51.781015 24898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:51.792491 24910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:51.792526 24905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:51.792914 24898 server_base.cc:1034] running on GCE node
W20260806 12:27:51.792491 24906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:51.793488 24898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:51.794288 24898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:51.795504 24898 hybrid_clock.cc:648] HybridClock initialized: now 1786019271795475 us; error 39 us; skew 500 ppm
Aug 06 12:27:51 dist-test-slave-lgr5 krb5kdc[24740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019271, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:51.798763 24898 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:27:51.799829 24898 webserver.cc:460] Webserver started at http://127.28.210.65:35115/ using document root <none> and password file <none>
I20260806 12:27:51.800506 24898 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:51.800585 24898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:51.800808 24898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:51.803088 24898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "12162e2d14104f32954c4ff30ec25f17"
format_stamp: "Formatted at 2026-08-06 12:27:51 on dist-test-slave-lgr5"
server_key: "fdc38d9ac5373c26c473978ded35d7cf"
server_key_iv: "850884f790867aeaa3db850d65ea6a64"
server_key_version: "encryptionkey@0"
I20260806 12:27:51.803696 24898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "12162e2d14104f32954c4ff30ec25f17"
format_stamp: "Formatted at 2026-08-06 12:27:51 on dist-test-slave-lgr5"
server_key: "fdc38d9ac5373c26c473978ded35d7cf"
server_key_iv: "850884f790867aeaa3db850d65ea6a64"
server_key_version: "encryptionkey@0"
I20260806 12:27:51.807916 24898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 12:27:51.810632 24917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:51.811964 24898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 12:27:51.812139 24898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "12162e2d14104f32954c4ff30ec25f17"
format_stamp: "Formatted at 2026-08-06 12:27:51 on dist-test-slave-lgr5"
server_key: "fdc38d9ac5373c26c473978ded35d7cf"
server_key_iv: "850884f790867aeaa3db850d65ea6a64"
server_key_version: "encryptionkey@0"
I20260806 12:27:51.812274 24898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:51.833964 24898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:51.837771 24898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:51.838009 24898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:51.838719 24898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:51.839674 24898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:51.839728 24898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:51.839785 24898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:51.839814 24898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:51.849291 24898 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:35859
I20260806 12:27:51.849315 25089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:35859 every 8 connection(s)
I20260806 12:27:51.850299 24898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:27:51.854780 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 24898
I20260806 12:27:51.854947 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:27:51.855342 29513 external_mini_cluster.cc:1428] Setting key d7e9a7b0ef1d160cee59bda7c71ffde5
Aug 06 12:27:51 dist-test-slave-lgr5 krb5kdc[24740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019271, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:27:51.867374 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:51.852346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:36683 (local address 127.28.210.126:46735)
0806 12:27:51.852515 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:51.852521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:51.853748 (+ 1227us) server_negotiation.cc:407] Connection header received
0806 12:27:51.855445 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:51.855449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:51.855512 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:51.855591 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:51.857220 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:51.857808 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:51.859601 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:51.859932 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:51.860050 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:51.863056 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:51.863060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:51.863063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:51.863098 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:51.864707 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:51.865454 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:51.865457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:51.865458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:51.865543 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:51.865969 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:51.865972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:51.865973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:51.866189 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:51.866267 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:51.867096 (+ 829us) server_negotiation.cc:299] Negotiation successful
0806 12:27:51.867230 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 12:27:51.868537 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:51.852913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:46735 (local address 127.28.210.65:36683)
0806 12:27:51.853552 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:51.853616 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:51.855133 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:51.855825 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:51.855836 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:51.856393 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:51.857053 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:51.857078 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:51.857953 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:51.857957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:51.859447 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:51.859462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:51.860160 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:51.860166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:51.860379 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:51.861048 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:51.861067 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:51.862837 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:51.864869 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:51.864878 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:51.864901 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:51.865341 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:51.865633 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:51.865637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:51.865638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:51.865866 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:51.866304 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:51.866312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:51.866752 (+ 440us) client_negotiation.cc:769] Sending connection context
0806 12:27:51.867014 (+ 262us) client_negotiation.cc:240] Negotiation successful
0806 12:27:51.867431 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":146,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:27:51.870146 25091 heartbeater.cc:347] Connected to a master server at 127.28.210.126:46735
I20260806 12:27:51.870474 25091 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:51.871069 25091 heartbeater.cc:511] Master 127.28.210.126:46735 requested a full tablet report, sending...
I20260806 12:27:51.872905 24806 ts_manager.cc:194] Registered new tserver with Master: 12162e2d14104f32954c4ff30ec25f17 (127.28.210.65:35859)
I20260806 12:27:51.874166 24806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:36683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:27:51.893507 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:46735
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:52.020752 25098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:52.021101 25098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:52.021170 25098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:52.025763 25098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:52.025861 25098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:52.025951 25098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:27:52.031874 25098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:46735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:52.033164 25098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:52.034406 25098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:52.045504 25105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:52.045621 25098 server_base.cc:1034] running on GCE node
W20260806 12:27:52.045483 25113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:52.045840 25108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:52.046293 25098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:52.047031 25098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:52.048189 25098 hybrid_clock.cc:648] HybridClock initialized: now 1786019272048168 us; error 35 us; skew 500 ppm
Aug 06 12:27:52 dist-test-slave-lgr5 krb5kdc[24740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:52.051195 25098 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:27:52.052193 25098 webserver.cc:460] Webserver started at http://127.28.210.66:44537/ using document root <none> and password file <none>
I20260806 12:27:52.052783 25098 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:52.052845 25098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:52.053031 25098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:52.055267 25098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "5cc0ac1d11dc416fa727a3042d7403e2"
format_stamp: "Formatted at 2026-08-06 12:27:52 on dist-test-slave-lgr5"
server_key: "4163a00e788ddaef4f0f15f7ada9536a"
server_key_iv: "ecb8eb069ba5bfd9b59285e7632dabae"
server_key_version: "encryptionkey@0"
I20260806 12:27:52.055907 25098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "5cc0ac1d11dc416fa727a3042d7403e2"
format_stamp: "Formatted at 2026-08-06 12:27:52 on dist-test-slave-lgr5"
server_key: "4163a00e788ddaef4f0f15f7ada9536a"
server_key_iv: "ecb8eb069ba5bfd9b59285e7632dabae"
server_key_version: "encryptionkey@0"
I20260806 12:27:52.060518 25098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 12:27:52.063542 25122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:52.065166 25098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:52.065318 25098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "5cc0ac1d11dc416fa727a3042d7403e2"
format_stamp: "Formatted at 2026-08-06 12:27:52 on dist-test-slave-lgr5"
server_key: "4163a00e788ddaef4f0f15f7ada9536a"
server_key_iv: "ecb8eb069ba5bfd9b59285e7632dabae"
server_key_version: "encryptionkey@0"
I20260806 12:27:52.065472 25098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:52.077627 25098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:52.081857 25098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:52.082084 25098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:52.082940 25098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:52.084215 25098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:52.084281 25098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:52.084474 25098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:52.084534 25098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:52.095841 25098 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:34315
I20260806 12:27:52.095872 25295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:34315 every 8 connection(s)
I20260806 12:27:52.097075 25098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:27:52.102041 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 25098
I20260806 12:27:52.102200 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:27:52.102558 29513 external_mini_cluster.cc:1428] Setting key 6b498a2452a7f0c565253fdd87837940
Aug 06 12:27:52 dist-test-slave-lgr5 krb5kdc[24740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
I20260806 12:27:52.114290 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.099316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:53429 (local address 127.28.210.126:46735)
0806 12:27:52.099492 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:52.099497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:52.100541 (+ 1044us) server_negotiation.cc:407] Connection header received
0806 12:27:52.102319 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:52.102323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:52.102393 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:52.102453 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:52.104315 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.104906 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.106607 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.106984 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.107119 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.110284 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:52.110288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:52.110290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:52.110319 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:52.111842 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:52.112469 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:52.112472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:52.112473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:52.112558 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:52.112925 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:52.112928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:52.112929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:52.113143 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:52.113236 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:52.113907 (+ 671us) server_negotiation.cc:299] Negotiation successful
0806 12:27:52.114098 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 12:27:52.115382 25299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.099751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:46735 (local address 127.28.210.66:53429)
0806 12:27:52.100366 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:52.100418 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:52.101982 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:52.102931 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:52.102941 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:52.103370 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:52.104135 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.104156 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.105080 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.105084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.106450 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.106464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.107321 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.107325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.107517 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.108281 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:52.108300 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:52.110060 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:52.111980 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:52.111987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:52.112007 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:52.112345 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:52.112648 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:52.112651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:52.112653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:52.112815 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:52.113257 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:52.113265 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:52.113581 (+ 316us) client_negotiation.cc:769] Sending connection context
0806 12:27:52.113766 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 12:27:52.114253 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":147,"threads_started":1}
I20260806 12:27:52.117017 25297 heartbeater.cc:347] Connected to a master server at 127.28.210.126:46735
I20260806 12:27:52.117304 25297 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:52.117939 25297 heartbeater.cc:511] Master 127.28.210.126:46735 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
I20260806 12:27:52.119432 24806 ts_manager.cc:194] Registered new tserver with Master: 5cc0ac1d11dc416fa727a3042d7403e2 (127.28.210.66:34315)
I20260806 12:27:52.120251 24806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:53429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:27:52.140007 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:46735
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:52.275446 25303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:52.275801 25303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:52.275871 25303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:52.280352 25303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:52.280434 25303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:52.280524 25303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:27:52.286768 25303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:46735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:52.288002 25303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:52.289149 25303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:52.299386 25313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:52.299389 25312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:52.299394 25320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:52.299718 25303 server_base.cc:1034] running on GCE node
I20260806 12:27:52.300228 25303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:52.300971 25303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:52.302239 25303 hybrid_clock.cc:648] HybridClock initialized: now 1786019272302198 us; error 41 us; skew 500 ppm
Aug 06 12:27:52 dist-test-slave-lgr5 krb5kdc[24740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:52.305816 25303 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:27:52.307026 25303 webserver.cc:460] Webserver started at http://127.28.210.67:46351/ using document root <none> and password file <none>
I20260806 12:27:52.307719 25303 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:52.307799 25303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:52.308053 25303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:52.310621 25303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "5267fedbc3f24e01a9420e674256d80e"
format_stamp: "Formatted at 2026-08-06 12:27:52 on dist-test-slave-lgr5"
server_key: "592728d795c51130953f61b389931c45"
server_key_iv: "3cec43e0ea5aeb990fdeb5b161a7979b"
server_key_version: "encryptionkey@0"
I20260806 12:27:52.311299 25303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "5267fedbc3f24e01a9420e674256d80e"
format_stamp: "Formatted at 2026-08-06 12:27:52 on dist-test-slave-lgr5"
server_key: "592728d795c51130953f61b389931c45"
server_key_iv: "3cec43e0ea5aeb990fdeb5b161a7979b"
server_key_version: "encryptionkey@0"
I20260806 12:27:52.315716 25303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 12:27:52.318856 25333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:52.320834 25303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:52.321019 25303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "5267fedbc3f24e01a9420e674256d80e"
format_stamp: "Formatted at 2026-08-06 12:27:52 on dist-test-slave-lgr5"
server_key: "592728d795c51130953f61b389931c45"
server_key_iv: "3cec43e0ea5aeb990fdeb5b161a7979b"
server_key_version: "encryptionkey@0"
I20260806 12:27:52.321187 25303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:52.350049 25303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:52.354012 25303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:52.354228 25303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:52.354925 25303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:52.355913 25303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:52.355965 25303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:52.356010 25303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:52.356040 25303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:52.367646 25303 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:40889
I20260806 12:27:52.367983 25504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:40889 every 8 connection(s)
I20260806 12:27:52.369208 25303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:27:52.376708 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 25303
I20260806 12:27:52.376873 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:27:52.377193 29513 external_mini_cluster.cc:1428] Setting key 730d02fdbfef3b1abf154b99a3b9366f
Aug 06 12:27:52 dist-test-slave-lgr5 krb5kdc[24740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:27:52.386721 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.371598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:35419 (local address 127.28.210.126:46735)
0806 12:27:52.371783 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:52.371787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:52.373059 (+ 1272us) server_negotiation.cc:407] Connection header received
0806 12:27:52.374741 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:52.374745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:52.374818 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:52.374915 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:52.376801 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.377507 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.379053 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.379302 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.379410 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.382568 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:52.382572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:52.382574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:52.382608 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:52.383924 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:52.384609 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:52.384612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:52.384613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:52.384699 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:52.385097 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:52.385101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:52.385102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:52.385377 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:52.385469 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:52.386447 (+ 978us) server_negotiation.cc:299] Negotiation successful
0806 12:27:52.386581 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 12:27:52.387651 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.372131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:46735 (local address 127.28.210.67:35419)
0806 12:27:52.372811 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:52.372873 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:52.374422 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:52.375184 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:52.375194 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:52.375716 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:52.376443 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.376464 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.377690 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.377694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.378895 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.378912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.379494 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.379499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.379707 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.380415 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:52.380436 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:52.382283 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:52.384128 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:52.384137 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:52.384160 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:52.384497 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:52.384771 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:52.384773 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:52.384775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:52.384959 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:52.385761 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:52.385769 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:52.386112 (+ 343us) client_negotiation.cc:769] Sending connection context
0806 12:27:52.386334 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 12:27:52.386665 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":164,"threads_started":1}
I20260806 12:27:52.389132 25505 heartbeater.cc:347] Connected to a master server at 127.28.210.126:46735
I20260806 12:27:52.389457 25505 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:52.390137 25505 heartbeater.cc:511] Master 127.28.210.126:46735 requested a full tablet report, sending...
I20260806 12:27:52.391568 24806 ts_manager.cc:194] Registered new tserver with Master: 5267fedbc3f24e01a9420e674256d80e (127.28.210.67:40889)
I20260806 12:27:52.392369 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:27:52.392370 24806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:35419
I20260806 12:27:52.403009 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.394883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59608 (local address 127.28.210.126:46735)
0806 12:27:52.395046 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:52.395052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:52.395266 (+ 214us) server_negotiation.cc:407] Connection header received
0806 12:27:52.395361 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:52.395363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:52.395413 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:52.395490 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:52.396384 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.396930 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.397888 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.398113 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.398198 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.399772 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:52.399776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:52.399778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:52.399803 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:52.401065 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:52.401597 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:52.401601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:52.401602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:52.401698 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:52.401962 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:52.401964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:52.401965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:52.402194 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:52.402364 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:52.402723 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 12:27:52.402833 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 12:27:52.406268 24806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:27:52.409443 24806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:27:52.426013 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.420733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.28.210.67:40889)
0806 12:27:52.421092 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:52.421103 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:52.421121 (+ 18us) server_negotiation.cc:407] Connection header received
0806 12:27:52.421185 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:52.421190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:52.421361 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:52.421527 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:52.421895 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.423132 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.424499 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.425223 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.425332 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.425514 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:52.425703 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 12:27:52.425829 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":145,"threads_started":1}
I20260806 12:27:52.426026 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.419414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:34315 (local address 127.0.0.1:46668)
0806 12:27:52.419885 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:52.419940 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:52.420107 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:52.421451 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:52.421458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:52.421486 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:52.421646 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.421653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.423292 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.423296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.424422 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.424431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.425527 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.425531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.425638 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.425662 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 12:27:52.425718 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 12:27:52.425894 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":78,"thread_start_us":117,"threads_started":1}
I20260806 12:27:52.426204 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.420170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:35859 (local address 127.0.0.1:50790)
0806 12:27:52.420450 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:52.420463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:52.420561 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:52.421523 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:52.421526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:52.421546 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:52.421670 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.421676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.422849 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.422852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.424222 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.424235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.425527 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.425531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.425864 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.425887 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 12:27:52.425972 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 12:27:52.426051 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":226,"thread_start_us":121,"threads_started":1}
I20260806 12:27:52.426254 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.420344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.28.210.65:35859)
0806 12:27:52.421062 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:52.421069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:52.421095 (+ 26us) server_negotiation.cc:407] Connection header received
0806 12:27:52.421174 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:52.421179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:52.421361 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:52.421826 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:52.421976 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.422714 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.424378 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.425348 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.425586 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.425884 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:52.426011 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 12:27:52.426115 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":93,"threads_started":1}
I20260806 12:27:52.426709 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.420279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:40889 (local address 127.0.0.1:37264)
0806 12:27:52.420755 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:52.420765 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:52.420829 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:52.421678 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:52.421681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:52.421696 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:52.421768 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.421774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.423327 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.423329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.424354 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.424366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.425350 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.425362 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.425486 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.425528 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 12:27:52.425648 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 12:27:52.426577 (+ 929us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260806 12:27:52.427024 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.419603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.28.210.66:34315)
0806 12:27:52.420739 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:52.420752 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:52.420779 (+ 27us) server_negotiation.cc:407] Connection header received
0806 12:27:52.420886 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:52.420891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:52.421128 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:52.421784 (+ 656us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:52.421960 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.423132 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.424553 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.425400 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.426568 (+ 1168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.426724 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:52.426774 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 12:27:52.426867 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015,"thread_start_us":154,"threads_started":1}
I20260806 12:27:52.429683 25409 tablet_service.cc:1467] Processing CreateTablet for tablet fd500cf3f2c74da98dc70e7eb4a84193 (DEFAULT_TABLE table=test-table [id=b29e3b30e1e041fb8232bc859b882ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:52.429682 24984 tablet_service.cc:1467] Processing CreateTablet for tablet fd500cf3f2c74da98dc70e7eb4a84193 (DEFAULT_TABLE table=test-table [id=b29e3b30e1e041fb8232bc859b882ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:52.430830 25409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd500cf3f2c74da98dc70e7eb4a84193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:52.430830 24984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd500cf3f2c74da98dc70e7eb4a84193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:52.435623 25202 tablet_service.cc:1467] Processing CreateTablet for tablet fd500cf3f2c74da98dc70e7eb4a84193 (DEFAULT_TABLE table=test-table [id=b29e3b30e1e041fb8232bc859b882ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:52.437098 25202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd500cf3f2c74da98dc70e7eb4a84193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:52.438582 25536 tablet_bootstrap.cc:492] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e: Bootstrap starting.
I20260806 12:27:52.439571 25537 tablet_bootstrap.cc:492] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17: Bootstrap starting.
I20260806 12:27:52.441107 25536 tablet_bootstrap.cc:654] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:52.441941 25537 tablet_bootstrap.cc:654] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:52.442430 25536 log.cc:824] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:52.442839 25537 log.cc:824] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:52.445412 25537 tablet_bootstrap.cc:492] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17: No bootstrap required, opened a new log
I20260806 12:27:52.445413 25536 tablet_bootstrap.cc:492] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e: No bootstrap required, opened a new log
I20260806 12:27:52.445672 25537 ts_tablet_manager.cc:1397] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 12:27:52.445672 25536 ts_tablet_manager.cc:1397] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 12:27:52.446033 25541 tablet_bootstrap.cc:492] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2: Bootstrap starting.
I20260806 12:27:52.448638 25541 tablet_bootstrap.cc:654] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:52.449915 25541 log.cc:824] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:52.450196 25536 raft_consensus.cc:348] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc0ac1d11dc416fa727a3042d7403e2" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34315 } } peers { permanent_uuid: "5267fedbc3f24e01a9420e674256d80e" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40889 } } peers { permanent_uuid: "12162e2d14104f32954c4ff30ec25f17" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 35859 } }
I20260806 12:27:52.450569 25536 raft_consensus.cc:374] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:52.450667 25536 raft_consensus.cc:729] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5267fedbc3f24e01a9420e674256d80e, State: Initialized, Role: FOLLOWER
I20260806 12:27:52.450553 25537 raft_consensus.cc:348] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc0ac1d11dc416fa727a3042d7403e2" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34315 } } peers { permanent_uuid: "5267fedbc3f24e01a9420e674256d80e" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40889 } } peers { permanent_uuid: "12162e2d14104f32954c4ff30ec25f17" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 35859 } }
I20260806 12:27:52.450915 25537 raft_consensus.cc:374] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:52.451014 25537 raft_consensus.cc:729] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12162e2d14104f32954c4ff30ec25f17, State: Initialized, Role: FOLLOWER
I20260806 12:27:52.451498 25537 consensus_queue.cc:260] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc0ac1d11dc416fa727a3042d7403e2" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34315 } } peers { permanent_uuid: "5267fedbc3f24e01a9420e674256d80e" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40889 } } peers { permanent_uuid: "12162e2d14104f32954c4ff30ec25f17" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 35859 } }
I20260806 12:27:52.451874 25536 consensus_queue.cc:260] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc0ac1d11dc416fa727a3042d7403e2" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34315 } } peers { permanent_uuid: "5267fedbc3f24e01a9420e674256d80e" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40889 } } peers { permanent_uuid: "12162e2d14104f32954c4ff30ec25f17" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 35859 } }
I20260806 12:27:52.452756 25541 tablet_bootstrap.cc:492] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2: No bootstrap required, opened a new log
I20260806 12:27:52.452934 25536 ts_tablet_manager.cc:1428] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 12:27:52.452934 25537 ts_tablet_manager.cc:1428] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260806 12:27:52.452992 25541 ts_tablet_manager.cc:1397] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 12:27:52.453415 25505 heartbeater.cc:503] Master 127.28.210.126:46735 was elected leader, sending a full tablet report...
I20260806 12:27:52.453791 25091 heartbeater.cc:503] Master 127.28.210.126:46735 was elected leader, sending a full tablet report...
I20260806 12:27:52.457113 25541 raft_consensus.cc:348] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc0ac1d11dc416fa727a3042d7403e2" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34315 } } peers { permanent_uuid: "5267fedbc3f24e01a9420e674256d80e" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40889 } } peers { permanent_uuid: "12162e2d14104f32954c4ff30ec25f17" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 35859 } }
I20260806 12:27:52.457504 25541 raft_consensus.cc:374] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:52.457569 25541 raft_consensus.cc:729] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc0ac1d11dc416fa727a3042d7403e2, State: Initialized, Role: FOLLOWER
I20260806 12:27:52.458052 25541 consensus_queue.cc:260] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc0ac1d11dc416fa727a3042d7403e2" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34315 } } peers { permanent_uuid: "5267fedbc3f24e01a9420e674256d80e" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40889 } } peers { permanent_uuid: "12162e2d14104f32954c4ff30ec25f17" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 35859 } }
I20260806 12:27:52.458876 25297 heartbeater.cc:503] Master 127.28.210.126:46735 was elected leader, sending a full tablet report...
I20260806 12:27:52.460451 25541 ts_tablet_manager.cc:1428] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260806 12:27:52.598548 25298 tablet.cc:2369] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:27:52.601909 25092 tablet.cc:2369] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:27:52.620445 25507 tablet.cc:2369] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:27:52.722844 25546 raft_consensus.cc:482] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:27:52.723086 25546 raft_consensus.cc:504] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc0ac1d11dc416fa727a3042d7403e2" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34315 } } peers { permanent_uuid: "5267fedbc3f24e01a9420e674256d80e" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40889 } } peers { permanent_uuid: "12162e2d14104f32954c4ff30ec25f17" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 35859 } }
I20260806 12:27:52.724315 25546 leader_election.cc:288] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cc0ac1d11dc416fa727a3042d7403e2 (127.28.210.66:34315), 5267fedbc3f24e01a9420e674256d80e (127.28.210.67:40889)
I20260806 12:27:52.728920 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.724677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:46351 (local address 127.28.210.66:34315)
0806 12:27:52.724795 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:52.724798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:52.725003 (+ 205us) server_negotiation.cc:407] Connection header received
0806 12:27:52.725264 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:52.725267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:52.725330 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:52.725415 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:52.725764 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.726578 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.727947 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.728363 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.728455 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.728513 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:52.728699 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 12:27:52.728773 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 12:27:52.728931 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.725157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:48779 (local address 127.28.210.67:40889)
0806 12:27:52.725267 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:52.725271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:52.725518 (+ 247us) server_negotiation.cc:407] Connection header received
0806 12:27:52.725704 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:52.725707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:52.725762 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:52.725835 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:52.726068 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.726773 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.727898 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.728399 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.728518 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.728570 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:52.728726 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 12:27:52.728800 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 12:27:52.728951 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.725044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:40889 (local address 127.28.210.65:48779)
0806 12:27:52.725398 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:52.725440 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:52.725571 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:52.725856 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:52.725859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:52.725875 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:52.725968 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.725974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.726887 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.726890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.727772 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.727783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.728531 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.728535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.728612 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.728630 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 12:27:52.728702 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 12:27:52.728768 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":33,"spinlock_wait_cycles":2816,"thread_start_us":106,"threads_started":1}
I20260806 12:27:52.728955 25553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.724602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:34315 (local address 127.28.210.65:46351)
0806 12:27:52.724910 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:52.724925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:52.725050 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:52.725455 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:52.725459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:52.725483 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:52.725655 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.725662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.726748 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.726751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.727805 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:52.727818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.728478 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.728482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:52.728576 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.728601 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 12:27:52.728663 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 12:27:52.728734 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":45,"spinlock_wait_cycles":14592,"thread_start_us":118,"threads_started":1}
I20260806 12:27:52.729687 25433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd500cf3f2c74da98dc70e7eb4a84193" candidate_uuid: "12162e2d14104f32954c4ff30ec25f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5267fedbc3f24e01a9420e674256d80e" is_pre_election: true
I20260806 12:27:52.729745 25237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd500cf3f2c74da98dc70e7eb4a84193" candidate_uuid: "12162e2d14104f32954c4ff30ec25f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc0ac1d11dc416fa727a3042d7403e2" is_pre_election: true
I20260806 12:27:52.730046 25433 raft_consensus.cc:2454] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12162e2d14104f32954c4ff30ec25f17 in term 0.
I20260806 12:27:52.730046 25237 raft_consensus.cc:2454] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12162e2d14104f32954c4ff30ec25f17 in term 0.
I20260806 12:27:52.730600 24921 leader_election.cc:302] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12162e2d14104f32954c4ff30ec25f17, 5cc0ac1d11dc416fa727a3042d7403e2; no voters:
I20260806 12:27:52.730870 25546 raft_consensus.cc:2781] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:27:52.730967 25546 raft_consensus.cc:482] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:27:52.731009 25546 raft_consensus.cc:3037] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:52.732194 25546 raft_consensus.cc:504] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc0ac1d11dc416fa727a3042d7403e2" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34315 } } peers { permanent_uuid: "5267fedbc3f24e01a9420e674256d80e" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40889 } } peers { permanent_uuid: "12162e2d14104f32954c4ff30ec25f17" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 35859 } }
I20260806 12:27:52.732564 25546 leader_election.cc:288] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [CANDIDATE]: Term 1 election: Requested vote from peers 5cc0ac1d11dc416fa727a3042d7403e2 (127.28.210.66:34315), 5267fedbc3f24e01a9420e674256d80e (127.28.210.67:40889)
I20260806 12:27:52.733172 25433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd500cf3f2c74da98dc70e7eb4a84193" candidate_uuid: "12162e2d14104f32954c4ff30ec25f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5267fedbc3f24e01a9420e674256d80e"
I20260806 12:27:52.733176 25237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd500cf3f2c74da98dc70e7eb4a84193" candidate_uuid: "12162e2d14104f32954c4ff30ec25f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc0ac1d11dc416fa727a3042d7403e2"
I20260806 12:27:52.733337 25237 raft_consensus.cc:3037] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:52.733359 25433 raft_consensus.cc:3037] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:52.734763 25433 raft_consensus.cc:2454] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12162e2d14104f32954c4ff30ec25f17 in term 1.
I20260806 12:27:52.734763 25237 raft_consensus.cc:2454] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12162e2d14104f32954c4ff30ec25f17 in term 1.
I20260806 12:27:52.735183 24921 leader_election.cc:302] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12162e2d14104f32954c4ff30ec25f17, 5cc0ac1d11dc416fa727a3042d7403e2; no voters:
I20260806 12:27:52.735392 25546 raft_consensus.cc:2781] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:52.735638 25546 raft_consensus.cc:686] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [term 1 LEADER]: Becoming Leader. State: Replica: 12162e2d14104f32954c4ff30ec25f17, State: Running, Role: LEADER
I20260806 12:27:52.736063 25546 consensus_queue.cc:237] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc0ac1d11dc416fa727a3042d7403e2" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34315 } } peers { permanent_uuid: "5267fedbc3f24e01a9420e674256d80e" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40889 } } peers { permanent_uuid: "12162e2d14104f32954c4ff30ec25f17" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 35859 } }
I20260806 12:27:52.739853 24803 catalog_manager.cc:5495] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12162e2d14104f32954c4ff30ec25f17 (127.28.210.65). New cstate: current_term: 1 leader_uuid: "12162e2d14104f32954c4ff30ec25f17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc0ac1d11dc416fa727a3042d7403e2" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5267fedbc3f24e01a9420e674256d80e" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12162e2d14104f32954c4ff30ec25f17" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 35859 } health_report { overall_health: HEALTHY } } }
I20260806 12:27:52.797426 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.793797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.28.210.65:35859)
0806 12:27:52.793966 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:52.793971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:52.793986 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:27:52.794073 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:52.794075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:52.794133 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:52.794232 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:27:52.794489 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.795002 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.795841 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:52.796065 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:52.796196 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:52.796538 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:27:52.796987 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:27:52.797122 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:52.797245 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 12:27:52.797290 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 12:27:52.803679 25433 raft_consensus.cc:1261] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e [term 1 FOLLOWER]: Refusing update from remote peer 12162e2d14104f32954c4ff30ec25f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:52.803716 25237 raft_consensus.cc:1261] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2 [term 1 FOLLOWER]: Refusing update from remote peer 12162e2d14104f32954c4ff30ec25f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:52.804447 25546 consensus_queue.cc:1035] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5267fedbc3f24e01a9420e674256d80e" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:52.804658 25556 consensus_queue.cc:1035] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cc0ac1d11dc416fa727a3042d7403e2" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 34315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:52.813663 25560 mvcc.cc:205] Tried to move back new op lower bound from 7315534941395828736 to 7315534941130633216. Current Snapshot: MvccSnapshot[applied={T|T < 7315534941395828736}]
I20260806 12:27:52.823858 24806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:52.824055 24806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:52.827023 24806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c95100b65d6149cab19d5c2a6178eeca: Sending DeleteTablet for 3 replicas of tablet fd500cf3f2c74da98dc70e7eb4a84193
I20260806 12:27:52.828107 25409 tablet_service.cc:1514] Processing DeleteTablet for tablet fd500cf3f2c74da98dc70e7eb4a84193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:52 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:37264
I20260806 12:27:52.828110 25202 tablet_service.cc:1514] Processing DeleteTablet for tablet fd500cf3f2c74da98dc70e7eb4a84193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:52 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:46668
I20260806 12:27:52.828176 24984 tablet_service.cc:1514] Processing DeleteTablet for tablet fd500cf3f2c74da98dc70e7eb4a84193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:52 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:50790
I20260806 12:27:52.828802 25578 tablet_replica.cc:330] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17: stopping tablet replica
I20260806 12:27:52.829211 25578 raft_consensus.cc:2229] T fd500cf3f2c74da98dc70e7eb4a84193 P 12162e2d14104f32954c4ff30ec25f17 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:27:52.829540 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 24898
I20260806 12:27:52.832896 25577 tablet_replica.cc:330] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2: stopping tablet replica
I20260806 12:27:52.833333 25577 raft_consensus.cc:2229] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:27:52.833688 25577 raft_consensus.cc:2258] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:52.834887 25576 tablet_replica.cc:330] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e: stopping tablet replica
I20260806 12:27:52.835314 25576 raft_consensus.cc:2229] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:27:52.835680 25576 raft_consensus.cc:2258] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:52.841396 25577 ts_tablet_manager.cc:1905] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:52.841396 25576 ts_tablet_manager.cc:1905] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:52.845026 25577 ts_tablet_manager.cc:1918] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:52.845026 25576 ts_tablet_manager.cc:1918] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:52.845203 25576 log.cc:1196] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/fd500cf3f2c74da98dc70e7eb4a84193
I20260806 12:27:52.845203 25577 log.cc:1196] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/fd500cf3f2c74da98dc70e7eb4a84193
I20260806 12:27:52.845605 25577 ts_tablet_manager.cc:1939] T fd500cf3f2c74da98dc70e7eb4a84193 P 5cc0ac1d11dc416fa727a3042d7403e2: Deleting consensus metadata
I20260806 12:27:52.845605 25576 ts_tablet_manager.cc:1939] T fd500cf3f2c74da98dc70e7eb4a84193 P 5267fedbc3f24e01a9420e674256d80e: Deleting consensus metadata
I20260806 12:27:52.847654 24782 catalog_manager.cc:4841] TS 5cc0ac1d11dc416fa727a3042d7403e2 (127.28.210.66:34315): tablet fd500cf3f2c74da98dc70e7eb4a84193 (table test-table [id=b29e3b30e1e041fb8232bc859b882ef5]) successfully deleted
I20260806 12:27:52.847952 24782 catalog_manager.cc:4841] TS 5267fedbc3f24e01a9420e674256d80e (127.28.210.67:40889): tablet fd500cf3f2c74da98dc70e7eb4a84193 (table test-table [id=b29e3b30e1e041fb8232bc859b882ef5]) successfully deleted
W20260806 12:27:52.848397 24783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:35859 (error 108)
I20260806 12:27:52.848780 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 25098
I20260806 12:27:52.852414 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:52.851918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:35859 (local address 127.0.0.1:50812)
0806 12:27:52.852220 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:52.852313 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
W20260806 12:27:52.852661 24783 catalog_manager.cc:4584] TS 12162e2d14104f32954c4ff30ec25f17 (127.28.210.65:35859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd500cf3f2c74da98dc70e7eb4a84193: Network error: Client connection negotiation failed: client connection to 127.28.210.65:35859: connect: Connection refused (error 111)
I20260806 12:27:52.868533 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 25303
I20260806 12:27:52.880771 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 24755
2026-08-06T12:27:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2943 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T12:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:27:52Z Disabled control of system clock
I20260806 12:27:52.922333 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:42689
--webserver_interface=127.28.210.126
--webserver_port=0
--builtin_ntp_servers=127.28.210.84:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:42689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 12:27:53.050822 25585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:53.051182 25585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:53.051223 25585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:53.056356 25585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:27:53.056463 25585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:53.056478 25585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:53.056495 25585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:27:53.056511 25585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:27:53.062053 25585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:42689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:42689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:53.063302 25585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:53.064779 25585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:53.074498 25592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:53.074499 25595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:53.074525 25585 server_base.cc:1034] running on GCE node
W20260806 12:27:53.074589 25598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:53.075443 25585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:53.076785 25585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:53.077980 25585 hybrid_clock.cc:648] HybridClock initialized: now 1786019273077952 us; error 43 us; skew 500 ppm
I20260806 12:27:53.080168 25585 webserver.cc:460] Webserver started at http://127.28.210.126:39307/ using document root <none> and password file <none>
I20260806 12:27:53.080921 25585 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:53.081090 25585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:53.081377 25585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:53.083832 25585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "d41be0fd134a4396b9551965b9c928a9"
format_stamp: "Formatted at 2026-08-06 12:27:53 on dist-test-slave-lgr5"
server_key: "81a701a8afc48a4ffe6bc0a2d26cab62"
server_key_iv: "090798ba6736844f9ff1ebb1e8420691"
server_key_version: "encryptionkey@0"
I20260806 12:27:53.084540 25585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "d41be0fd134a4396b9551965b9c928a9"
format_stamp: "Formatted at 2026-08-06 12:27:53 on dist-test-slave-lgr5"
server_key: "81a701a8afc48a4ffe6bc0a2d26cab62"
server_key_iv: "090798ba6736844f9ff1ebb1e8420691"
server_key_version: "encryptionkey@0"
I20260806 12:27:53.089947 25585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 12:27:53.093318 25605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:53.095167 25585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 12:27:53.095345 25585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "d41be0fd134a4396b9551965b9c928a9"
format_stamp: "Formatted at 2026-08-06 12:27:53 on dist-test-slave-lgr5"
server_key: "81a701a8afc48a4ffe6bc0a2d26cab62"
server_key_iv: "090798ba6736844f9ff1ebb1e8420691"
server_key_version: "encryptionkey@0"
I20260806 12:27:53.095496 25585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:53.114934 25585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:53.115882 25585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:53.116163 25585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:53.124437 25585 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:42689
I20260806 12:27:53.124472 25695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:42689 every 8 connection(s)
I20260806 12:27:53.125546 25585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:27:53.129081 25700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:53.131428 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 25585
I20260806 12:27:53.131590 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:27:53.131884 29513 external_mini_cluster.cc:1428] Setting key ab8d2b8285eea065d441ea88f8468148
I20260806 12:27:53.136144 25700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9: Bootstrap starting.
I20260806 12:27:53.139241 25700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:53.140280 25700 log.cc:824] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:53.142841 25700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9: No bootstrap required, opened a new log
I20260806 12:27:53.142935 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.133607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.28.210.126:42689)
0806 12:27:53.134298 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:53.134317 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:53.134358 (+ 41us) server_negotiation.cc:407] Connection header received
0806 12:27:53.135243 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:53.135268 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:53.135729 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:53.136063 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:53.136560 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.137613 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.138870 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.139568 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.139706 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.140732 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:53.140742 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:53.140764 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:53.141360 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:53.141429 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:53.141525 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:53.141631 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:53.141790 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 12:27:53.142132 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":131,"threads_started":1}
I20260806 12:27:53.148226 25700 raft_consensus.cc:348] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41be0fd134a4396b9551965b9c928a9" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42689 } }
I20260806 12:27:53.148598 25700 raft_consensus.cc:374] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:53.148705 25700 raft_consensus.cc:729] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41be0fd134a4396b9551965b9c928a9, State: Initialized, Role: FOLLOWER
I20260806 12:27:53.149222 25700 consensus_queue.cc:260] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41be0fd134a4396b9551965b9c928a9" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42689 } }
I20260806 12:27:53.149400 25700 raft_consensus.cc:388] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:27:53.149452 25700 raft_consensus.cc:482] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:27:53.149540 25700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:53.150708 25700 raft_consensus.cc:504] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41be0fd134a4396b9551965b9c928a9" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42689 } }
I20260806 12:27:53.151103 25700 leader_election.cc:302] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d41be0fd134a4396b9551965b9c928a9; no voters:
I20260806 12:27:53.151455 25700 leader_election.cc:288] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:27:53.151690 25706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:53.151990 25706 raft_consensus.cc:686] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [term 1 LEADER]: Becoming Leader. State: Replica: d41be0fd134a4396b9551965b9c928a9, State: Running, Role: LEADER
I20260806 12:27:53.152417 25706 consensus_queue.cc:237] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41be0fd134a4396b9551965b9c928a9" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42689 } }
I20260806 12:27:53.153070 25700 sys_catalog.cc:564] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:27:53.154373 25708 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d41be0fd134a4396b9551965b9c928a9. Latest consensus state: current_term: 1 leader_uuid: "d41be0fd134a4396b9551965b9c928a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41be0fd134a4396b9551965b9c928a9" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42689 } } }
I20260806 12:27:53.154670 25708 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:53.154386 25707 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d41be0fd134a4396b9551965b9c928a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41be0fd134a4396b9551965b9c928a9" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42689 } } }
I20260806 12:27:53.154804 25707 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9 [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:53.155220 25720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:27:53.158939 25720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:27:53.167624 25720 catalog_manager.cc:1347] Generated new cluster ID: 2f7fa4eed60f49af89dd812df13975ff
I20260806 12:27:53.167752 25720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:27:53.184502 25720 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:27:53.185665 25720 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:27:53.195649 25720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9: Generated new TSK 0
I20260806 12:27:53.196784 25720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:27:53.208494 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42689
--builtin_ntp_servers=127.28.210.84:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 12:27:53.340618 25735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:53.340983 25735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:53.341050 25735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:53.346362 25735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:53.346787 25735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:53.346944 25735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:27:53.354347 25735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:53.355791 25735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:53.357354 25735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:53.368904 25746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:53.368909 25744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:53.368909 25743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:53.369618 25735 server_base.cc:1034] running on GCE node
I20260806 12:27:53.370160 25735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:53.371025 25735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:53.372200 25735 hybrid_clock.cc:648] HybridClock initialized: now 1786019273372161 us; error 58 us; skew 500 ppm
I20260806 12:27:53.374723 25735 webserver.cc:460] Webserver started at http://127.28.210.65:41497/ using document root <none> and password file <none>
I20260806 12:27:53.375579 25735 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:53.375687 25735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:53.375953 25735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:53.378899 25735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "895956dd4ec4482b8edbd98fcf82a35d"
format_stamp: "Formatted at 2026-08-06 12:27:53 on dist-test-slave-lgr5"
server_key: "64149f6a7fe473625c98a00be020b854"
server_key_iv: "250a12b105b44e7739aeedf6c8fa3519"
server_key_version: "encryptionkey@0"
I20260806 12:27:53.379662 25735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "895956dd4ec4482b8edbd98fcf82a35d"
format_stamp: "Formatted at 2026-08-06 12:27:53 on dist-test-slave-lgr5"
server_key: "64149f6a7fe473625c98a00be020b854"
server_key_iv: "250a12b105b44e7739aeedf6c8fa3519"
server_key_version: "encryptionkey@0"
I20260806 12:27:53.384541 25735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 12:27:53.387629 25757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:53.389179 25735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:53.389323 25735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "895956dd4ec4482b8edbd98fcf82a35d"
format_stamp: "Formatted at 2026-08-06 12:27:53 on dist-test-slave-lgr5"
server_key: "64149f6a7fe473625c98a00be020b854"
server_key_iv: "250a12b105b44e7739aeedf6c8fa3519"
server_key_version: "encryptionkey@0"
I20260806 12:27:53.389499 25735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:53.408699 25735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:53.409597 25735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:53.409806 25735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:53.410547 25735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:53.411974 25735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:53.412047 25735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:53.412135 25735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:53.412158 25735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:53.424429 25735 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:44411
I20260806 12:27:53.424492 25930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:44411 every 8 connection(s)
I20260806 12:27:53.425650 25735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:27:53.429095 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 25735
I20260806 12:27:53.429247 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:27:53.429567 29513 external_mini_cluster.cc:1428] Setting key 4e3eb54055ce594876b28a21ca0a927e
I20260806 12:27:53.431771 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42689
--builtin_ntp_servers=127.28.210.84:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 12:27:53.438652 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.428493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:56401 (local address 127.28.210.126:42689)
0806 12:27:53.428875 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:53.428881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:53.430167 (+ 1286us) server_negotiation.cc:407] Connection header received
0806 12:27:53.431879 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:53.431883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:53.431949 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:53.432046 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:53.433229 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.433984 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.435950 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.436285 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.436388 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.438067 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:53.438070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:53.438072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:53.438091 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:53.438116 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:53.438125 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:53.438198 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:53.438397 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 12:27:53.438475 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260806 12:27:53.440830 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.429025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:42689 (local address 127.28.210.65:56401)
0806 12:27:53.429963 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:53.430035 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:53.431616 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:53.432300 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:53.432309 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:53.432416 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:53.433031 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:53.433066 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.434166 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.434170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:53.435757 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:53.435776 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.436508 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.436512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:53.436756 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.437627 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:27:53.437629 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:53.437702 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:27:53.437820 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:53.438225 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:53.438236 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:53.438244 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 12:27:53.438410 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 12:27:53.438855 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":195,"threads_started":1}
I20260806 12:27:53.442631 25932 heartbeater.cc:347] Connected to a master server at 127.28.210.126:42689
I20260806 12:27:53.442946 25932 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:53.443655 25932 heartbeater.cc:511] Master 127.28.210.126:42689 requested a full tablet report, sending...
I20260806 12:27:53.445843 25638 ts_manager.cc:194] Registered new tserver with Master: 895956dd4ec4482b8edbd98fcf82a35d (127.28.210.65:44411)
I20260806 12:27:53.447701 25638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.210.65:56401
W20260806 12:27:53.565238 25936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:53.565524 25936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:53.565562 25936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:53.569655 25936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:53.569720 25936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:53.569778 25936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:27:53.574801 25936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:53.576150 25936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:53.577831 25936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:53.589166 25943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:53.589149 25948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:53.589149 25944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:53.589210 25936 server_base.cc:1034] running on GCE node
I20260806 12:27:53.589859 25936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:53.590552 25936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:53.591737 25936 hybrid_clock.cc:648] HybridClock initialized: now 1786019273591707 us; error 38 us; skew 500 ppm
I20260806 12:27:53.593838 25936 webserver.cc:460] Webserver started at http://127.28.210.66:42057/ using document root <none> and password file <none>
I20260806 12:27:53.594483 25936 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:53.594556 25936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:53.594774 25936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:53.597018 25936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "a682b8cf6651478dac165186e061b5c7"
format_stamp: "Formatted at 2026-08-06 12:27:53 on dist-test-slave-lgr5"
server_key: "f834dd21a3e18be91d0c23ede056ed1a"
server_key_iv: "11c201d1298e1a522a18ba1a5502106f"
server_key_version: "encryptionkey@0"
I20260806 12:27:53.597587 25936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "a682b8cf6651478dac165186e061b5c7"
format_stamp: "Formatted at 2026-08-06 12:27:53 on dist-test-slave-lgr5"
server_key: "f834dd21a3e18be91d0c23ede056ed1a"
server_key_iv: "11c201d1298e1a522a18ba1a5502106f"
server_key_version: "encryptionkey@0"
I20260806 12:27:53.601493 25936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:27:53.604000 25956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:53.605294 25936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 12:27:53.605422 25936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "a682b8cf6651478dac165186e061b5c7"
format_stamp: "Formatted at 2026-08-06 12:27:53 on dist-test-slave-lgr5"
server_key: "f834dd21a3e18be91d0c23ede056ed1a"
server_key_iv: "11c201d1298e1a522a18ba1a5502106f"
server_key_version: "encryptionkey@0"
I20260806 12:27:53.605531 25936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:53.624823 25936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:53.625643 25936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:53.625864 25936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:53.626543 25936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:53.627609 25936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:53.627658 25936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:53.627717 25936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:53.627745 25936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:53.636758 25936 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:42463
I20260806 12:27:53.636783 26133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:42463 every 8 connection(s)
I20260806 12:27:53.637796 25936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:27:53.642055 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 25936
I20260806 12:27:53.642184 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:27:53.642498 29513 external_mini_cluster.cc:1428] Setting key d21ef70b89cba1c3372609c7ca7cc730
I20260806 12:27:53.644481 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42689
--builtin_ntp_servers=127.28.210.84:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 12:27:53.648056 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.639882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:45681 (local address 127.28.210.126:42689)
0806 12:27:53.640029 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:53.640033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:53.641254 (+ 1221us) server_negotiation.cc:407] Connection header received
0806 12:27:53.642861 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:53.642865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:53.642925 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:53.643002 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:53.644200 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.644778 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.646160 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.646381 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.646473 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.647579 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:53.647583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:53.647585 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:53.647601 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:53.647619 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:53.647627 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:53.647685 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:53.647835 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 12:27:53.647901 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 12:27:53.649075 26136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.640317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:42689 (local address 127.28.210.66:45681)
0806 12:27:53.641101 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:53.641145 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:53.642650 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:53.643365 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:53.643371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:53.643457 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:53.644040 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:53.644055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.644904 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.644907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:53.645993 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:53.646004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.646534 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.646538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:53.646704 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.647309 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:27:53.647310 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:53.647354 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:27:53.647428 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:53.647697 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:53.647707 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:53.647713 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 12:27:53.647823 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 12:27:53.648177 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":149,"threads_started":1}
I20260806 12:27:53.650431 26134 heartbeater.cc:347] Connected to a master server at 127.28.210.126:42689
I20260806 12:27:53.650688 26134 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:53.651283 26134 heartbeater.cc:511] Master 127.28.210.126:42689 requested a full tablet report, sending...
I20260806 12:27:53.652534 25638 ts_manager.cc:194] Registered new tserver with Master: a682b8cf6651478dac165186e061b5c7 (127.28.210.66:42463)
I20260806 12:27:53.653407 25638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.210.66:45681
W20260806 12:27:53.769397 26137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:53.769719 26137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:53.769759 26137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:53.773964 26137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:53.774055 26137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:53.774114 26137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:27:53.779469 26137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:53.780799 26137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:53.782426 26137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:53.793480 26150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:53.793470 26144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:53.793565 26137 server_base.cc:1034] running on GCE node
W20260806 12:27:53.793470 26146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:53.794265 26137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:53.795042 26137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:53.796195 26137 hybrid_clock.cc:648] HybridClock initialized: now 1786019273796169 us; error 41 us; skew 500 ppm
I20260806 12:27:53.798300 26137 webserver.cc:460] Webserver started at http://127.28.210.67:41021/ using document root <none> and password file <none>
I20260806 12:27:53.798905 26137 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:53.798971 26137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:53.799166 26137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:53.801626 26137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "84db2ce2c6154ec088190c1dce356b0f"
format_stamp: "Formatted at 2026-08-06 12:27:53 on dist-test-slave-lgr5"
server_key: "49de6946f441402bfeabf15d98f42b46"
server_key_iv: "636357d6740ad55817d1961be558ed0b"
server_key_version: "encryptionkey@0"
I20260806 12:27:53.802227 26137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "84db2ce2c6154ec088190c1dce356b0f"
format_stamp: "Formatted at 2026-08-06 12:27:53 on dist-test-slave-lgr5"
server_key: "49de6946f441402bfeabf15d98f42b46"
server_key_iv: "636357d6740ad55817d1961be558ed0b"
server_key_version: "encryptionkey@0"
I20260806 12:27:53.806612 26137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:27:53.809775 26156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:53.811255 26137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 12:27:53.811422 26137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "84db2ce2c6154ec088190c1dce356b0f"
format_stamp: "Formatted at 2026-08-06 12:27:53 on dist-test-slave-lgr5"
server_key: "49de6946f441402bfeabf15d98f42b46"
server_key_iv: "636357d6740ad55817d1961be558ed0b"
server_key_version: "encryptionkey@0"
I20260806 12:27:53.811556 26137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:53.832834 26137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:53.833711 26137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:53.833917 26137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:53.834626 26137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:53.835880 26137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:53.835938 26137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:53.836010 26137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:53.836040 26137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:53.845889 26137 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:41677
I20260806 12:27:53.845925 26329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:41677 every 8 connection(s)
I20260806 12:27:53.846927 26137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:27:53.852970 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 26137
I20260806 12:27:53.853127 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:27:53.853463 29513 external_mini_cluster.cc:1428] Setting key 63f4436cde6b6a01d481db77b2de016c
I20260806 12:27:53.857247 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.848996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:37129 (local address 127.28.210.126:42689)
0806 12:27:53.849186 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:53.849191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:53.850318 (+ 1127us) server_negotiation.cc:407] Connection header received
0806 12:27:53.851720 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:53.851722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:53.851780 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:53.851859 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:53.852963 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.853784 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.855333 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.855615 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.855729 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.856857 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:53.856860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:53.856863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:53.856875 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:53.856895 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:53.856903 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:53.856962 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:53.857067 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 12:27:53.857123 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 12:27:53.858165 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.849435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:42689 (local address 127.28.210.67:37129)
0806 12:27:53.850160 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:53.850205 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:53.851506 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:53.852053 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:53.852060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:53.852193 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:53.852785 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:53.852802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.853936 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.853940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:53.855195 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:53.855207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.855788 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.855792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:53.855953 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.856589 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:27:53.856591 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:53.856637 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:27:53.856711 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:53.856961 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:53.856966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:53.856970 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 12:27:53.857062 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 12:27:53.857311 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":125,"threads_started":1}
I20260806 12:27:53.859292 26330 heartbeater.cc:347] Connected to a master server at 127.28.210.126:42689
I20260806 12:27:53.859529 26330 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:53.860128 26330 heartbeater.cc:511] Master 127.28.210.126:42689 requested a full tablet report, sending...
I20260806 12:27:53.861264 25638 ts_manager.cc:194] Registered new tserver with Master: 84db2ce2c6154ec088190c1dce356b0f (127.28.210.67:41677)
I20260806 12:27:53.861946 25638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.210.67:37129
I20260806 12:27:53.868201 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:27:53.874482 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.870413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.28.210.126:42689)
0806 12:27:53.870582 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:53.870588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:53.870721 (+ 133us) server_negotiation.cc:407] Connection header received
0806 12:27:53.870856 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:53.870858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:53.870903 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:53.870972 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:53.871281 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.871827 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.872756 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.872972 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.873064 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.873973 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:53.873976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:53.873978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:53.873991 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:53.874012 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:53.874020 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:53.874119 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:53.874286 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 12:27:53.874331 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 12:27:53.877756 25638 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:27:53.880975 25638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:27:53.896373 26349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.890559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:41677 (local address 127.0.0.1:54266)
0806 12:27:53.890983 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:53.891035 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:53.891172 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:53.891822 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:53.891829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:53.891857 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:53.892000 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:53.892006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.893478 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.893481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:53.894481 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:53.894494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.895662 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.895666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:53.895785 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.895832 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 12:27:53.896001 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 12:27:53.896182 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":104,"threads_started":1}
I20260806 12:27:53.897046 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.891384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:42463 (local address 127.0.0.1:55926)
0806 12:27:53.891628 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:53.891640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:53.891723 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:53.893008 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:53.893012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:53.893032 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:53.893141 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:53.893147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.894534 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.894537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:53.895670 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:53.895681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.896638 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.896642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:53.896742 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.896770 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 12:27:53.896821 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 12:27:53.896903 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":102,"thread_start_us":110,"threads_started":1}
I20260806 12:27:53.897531 26351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.890743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.28.210.67:41677)
0806 12:27:53.891279 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:53.891291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:53.891315 (+ 24us) server_negotiation.cc:407] Connection header received
0806 12:27:53.891394 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:53.891398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:53.891614 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:53.892175 (+ 561us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:53.892326 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.893378 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.894602 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.895522 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.896815 (+ 1293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.896990 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:53.897037 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 12:27:53.897273 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":169,"threads_started":1}
I20260806 12:27:53.897559 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.891755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:44411 (local address 127.0.0.1:46060)
0806 12:27:53.891988 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:53.892000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:53.892109 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:53.893188 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:53.893192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:53.893211 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:53.893298 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:53.893304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.894786 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.894790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:53.896109 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:53.896120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.897124 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.897127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:53.897215 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.897238 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 12:27:53.897292 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 12:27:53.897440 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":26,"thread_start_us":92,"threads_started":1}
I20260806 12:27:53.897735 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.892356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.28.210.65:44411)
0806 12:27:53.892748 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:53.892754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:53.892777 (+ 23us) server_negotiation.cc:407] Connection header received
0806 12:27:53.892861 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:53.892866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:53.893042 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:53.893446 (+ 404us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:53.893588 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.894664 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.896202 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.896999 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.897110 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.897294 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:53.897454 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 12:27:53.897565 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":98,"threads_started":1}
I20260806 12:27:53.897804 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:53.891558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.28.210.66:42463)
0806 12:27:53.892482 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:53.892494 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:53.892519 (+ 25us) server_negotiation.cc:407] Connection header received
0806 12:27:53.892601 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:53.892607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:53.892823 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:53.893067 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:53.893227 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.894420 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.895791 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:53.896500 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:53.897236 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:53.897515 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:53.897565 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 12:27:53.897656 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":145,"threads_started":1}
I20260806 12:27:53.900928 25829 tablet_service.cc:1467] Processing CreateTablet for tablet 60ad38ba318e4d32970016505d29a6e7 (DEFAULT_TABLE table=test-table [id=ed1b10adaee542e594c6a1f437c2c22c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:53.900930 26032 tablet_service.cc:1467] Processing CreateTablet for tablet 60ad38ba318e4d32970016505d29a6e7 (DEFAULT_TABLE table=test-table [id=ed1b10adaee542e594c6a1f437c2c22c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:53.900930 26232 tablet_service.cc:1467] Processing CreateTablet for tablet 60ad38ba318e4d32970016505d29a6e7 (DEFAULT_TABLE table=test-table [id=ed1b10adaee542e594c6a1f437c2c22c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:53.902115 25829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ad38ba318e4d32970016505d29a6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:53.902129 26032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ad38ba318e4d32970016505d29a6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:53.902129 26232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60ad38ba318e4d32970016505d29a6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:53.910768 26363 tablet_bootstrap.cc:492] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d: Bootstrap starting.
I20260806 12:27:53.912012 26365 tablet_bootstrap.cc:492] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f: Bootstrap starting.
I20260806 12:27:53.912005 26364 tablet_bootstrap.cc:492] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7: Bootstrap starting.
I20260806 12:27:53.914173 26363 tablet_bootstrap.cc:654] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:53.914913 26364 tablet_bootstrap.cc:654] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:53.914913 26365 tablet_bootstrap.cc:654] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:53.915937 26365 log.cc:824] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:53.915937 26364 log.cc:824] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:53.915969 26363 log.cc:824] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:53.918640 26364 tablet_bootstrap.cc:492] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7: No bootstrap required, opened a new log
I20260806 12:27:53.918659 26363 tablet_bootstrap.cc:492] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d: No bootstrap required, opened a new log
I20260806 12:27:53.918659 26365 tablet_bootstrap.cc:492] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f: No bootstrap required, opened a new log
I20260806 12:27:53.918884 26363 ts_tablet_manager.cc:1397] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 12:27:53.918884 26365 ts_tablet_manager.cc:1397] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 12:27:53.918884 26364 ts_tablet_manager.cc:1397] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 12:27:53.922751 26364 raft_consensus.cc:348] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84db2ce2c6154ec088190c1dce356b0f" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41677 } } peers { permanent_uuid: "a682b8cf6651478dac165186e061b5c7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42463 } } peers { permanent_uuid: "895956dd4ec4482b8edbd98fcf82a35d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44411 } }
I20260806 12:27:53.923125 26364 raft_consensus.cc:374] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:53.923197 26364 raft_consensus.cc:729] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a682b8cf6651478dac165186e061b5c7, State: Initialized, Role: FOLLOWER
I20260806 12:27:53.923434 26365 raft_consensus.cc:348] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84db2ce2c6154ec088190c1dce356b0f" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41677 } } peers { permanent_uuid: "a682b8cf6651478dac165186e061b5c7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42463 } } peers { permanent_uuid: "895956dd4ec4482b8edbd98fcf82a35d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44411 } }
I20260806 12:27:53.923527 26363 raft_consensus.cc:348] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84db2ce2c6154ec088190c1dce356b0f" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41677 } } peers { permanent_uuid: "a682b8cf6651478dac165186e061b5c7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42463 } } peers { permanent_uuid: "895956dd4ec4482b8edbd98fcf82a35d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44411 } }
I20260806 12:27:53.923720 26365 raft_consensus.cc:374] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:53.923741 26363 raft_consensus.cc:374] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:53.923774 26365 raft_consensus.cc:729] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84db2ce2c6154ec088190c1dce356b0f, State: Initialized, Role: FOLLOWER
I20260806 12:27:53.923784 26363 raft_consensus.cc:729] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895956dd4ec4482b8edbd98fcf82a35d, State: Initialized, Role: FOLLOWER
I20260806 12:27:53.923719 26364 consensus_queue.cc:260] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84db2ce2c6154ec088190c1dce356b0f" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41677 } } peers { permanent_uuid: "a682b8cf6651478dac165186e061b5c7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42463 } } peers { permanent_uuid: "895956dd4ec4482b8edbd98fcf82a35d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44411 } }
I20260806 12:27:53.924260 26363 consensus_queue.cc:260] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84db2ce2c6154ec088190c1dce356b0f" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41677 } } peers { permanent_uuid: "a682b8cf6651478dac165186e061b5c7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42463 } } peers { permanent_uuid: "895956dd4ec4482b8edbd98fcf82a35d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44411 } }
I20260806 12:27:53.924260 26365 consensus_queue.cc:260] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84db2ce2c6154ec088190c1dce356b0f" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41677 } } peers { permanent_uuid: "a682b8cf6651478dac165186e061b5c7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42463 } } peers { permanent_uuid: "895956dd4ec4482b8edbd98fcf82a35d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44411 } }
I20260806 12:27:53.924775 26134 heartbeater.cc:503] Master 127.28.210.126:42689 was elected leader, sending a full tablet report...
I20260806 12:27:53.925036 26364 ts_tablet_manager.cc:1428] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260806 12:27:53.925282 26365 ts_tablet_manager.cc:1428] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 12:27:53.925288 26363 ts_tablet_manager.cc:1428] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260806 12:27:53.925623 25932 heartbeater.cc:503] Master 127.28.210.126:42689 was elected leader, sending a full tablet report...
I20260806 12:27:53.925856 26330 heartbeater.cc:503] Master 127.28.210.126:42689 was elected leader, sending a full tablet report...
W20260806 12:27:53.927100 25933 tablet.cc:2369] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:27:54.060009 26370 raft_consensus.cc:482] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:27:54.060293 26370 raft_consensus.cc:504] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84db2ce2c6154ec088190c1dce356b0f" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41677 } } peers { permanent_uuid: "a682b8cf6651478dac165186e061b5c7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42463 } } peers { permanent_uuid: "895956dd4ec4482b8edbd98fcf82a35d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44411 } }
I20260806 12:27:54.061625 26370 leader_election.cc:288] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84db2ce2c6154ec088190c1dce356b0f (127.28.210.67:41677), 895956dd4ec4482b8edbd98fcf82a35d (127.28.210.65:44411)
I20260806 12:27:54.066310 26351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:54.062053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:41683 (local address 127.28.210.67:41677)
0806 12:27:54.062191 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:54.062196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:54.062218 (+ 22us) server_negotiation.cc:407] Connection header received
0806 12:27:54.062288 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:54.062291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:54.062362 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:54.062459 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:54.062979 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:54.063636 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:54.065039 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:54.065683 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:54.065790 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:54.065846 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:54.066113 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 12:27:54.066187 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 12:27:54.066334 26136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:54.061913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:41677 (local address 127.28.210.66:41683)
0806 12:27:54.062049 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:54.062065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:54.062170 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:54.062591 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:54.062594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:54.062616 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:54.062731 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:54.062738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:54.063773 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:54.063777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:54.064872 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:54.064883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:54.065842 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:54.065846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:54.065957 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:54.065989 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 12:27:54.066082 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 12:27:54.066173 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 12:27:54.067190 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:54.062363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:49739 (local address 127.28.210.65:44411)
0806 12:27:54.062527 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:54.062532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:54.062733 (+ 201us) server_negotiation.cc:407] Connection header received
0806 12:27:54.062928 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:54.062932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:54.062985 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:54.063044 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:54.063414 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:54.064239 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:54.065823 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:54.066446 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:54.066579 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:54.066632 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:54.066997 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 12:27:54.067081 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 12:27:54.067202 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:54.062195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:44411 (local address 127.28.210.66:49739)
0806 12:27:54.062642 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:54.062654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:54.062749 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:54.063101 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:54.063104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:54.063126 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:54.063259 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:54.063265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:54.064386 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:54.064390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:54.065508 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:54.065528 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:54.066752 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:54.066756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:54.066852 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:54.066879 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 12:27:54.066960 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 12:27:54.067042 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":201,"threads_started":1}
I20260806 12:27:54.067193 26263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ad38ba318e4d32970016505d29a6e7" candidate_uuid: "a682b8cf6651478dac165186e061b5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84db2ce2c6154ec088190c1dce356b0f" is_pre_election: true
I20260806 12:27:54.067597 26263 raft_consensus.cc:2454] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a682b8cf6651478dac165186e061b5c7 in term 0.
I20260806 12:27:54.067996 25859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ad38ba318e4d32970016505d29a6e7" candidate_uuid: "a682b8cf6651478dac165186e061b5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895956dd4ec4482b8edbd98fcf82a35d" is_pre_election: true
I20260806 12:27:54.068233 25957 leader_election.cc:302] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84db2ce2c6154ec088190c1dce356b0f, a682b8cf6651478dac165186e061b5c7; no voters:
I20260806 12:27:54.068351 25859 raft_consensus.cc:2454] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a682b8cf6651478dac165186e061b5c7 in term 0.
I20260806 12:27:54.068670 26370 raft_consensus.cc:2781] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:27:54.068795 26370 raft_consensus.cc:482] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:27:54.068848 26370 raft_consensus.cc:3037] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:54.070010 26370 raft_consensus.cc:504] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84db2ce2c6154ec088190c1dce356b0f" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41677 } } peers { permanent_uuid: "a682b8cf6651478dac165186e061b5c7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42463 } } peers { permanent_uuid: "895956dd4ec4482b8edbd98fcf82a35d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44411 } }
I20260806 12:27:54.070416 26370 leader_election.cc:288] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [CANDIDATE]: Term 1 election: Requested vote from peers 84db2ce2c6154ec088190c1dce356b0f (127.28.210.67:41677), 895956dd4ec4482b8edbd98fcf82a35d (127.28.210.65:44411)
I20260806 12:27:54.070902 25859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ad38ba318e4d32970016505d29a6e7" candidate_uuid: "a682b8cf6651478dac165186e061b5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895956dd4ec4482b8edbd98fcf82a35d"
I20260806 12:27:54.070902 26263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60ad38ba318e4d32970016505d29a6e7" candidate_uuid: "a682b8cf6651478dac165186e061b5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84db2ce2c6154ec088190c1dce356b0f"
I20260806 12:27:54.071053 25859 raft_consensus.cc:3037] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:54.071053 26263 raft_consensus.cc:3037] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:54.072471 26263 raft_consensus.cc:2454] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a682b8cf6651478dac165186e061b5c7 in term 1.
I20260806 12:27:54.072471 25859 raft_consensus.cc:2454] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a682b8cf6651478dac165186e061b5c7 in term 1.
I20260806 12:27:54.072826 25957 leader_election.cc:302] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84db2ce2c6154ec088190c1dce356b0f, a682b8cf6651478dac165186e061b5c7; no voters:
I20260806 12:27:54.073174 26370 raft_consensus.cc:2781] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:54.073599 26370 raft_consensus.cc:686] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 1 LEADER]: Becoming Leader. State: Replica: a682b8cf6651478dac165186e061b5c7, State: Running, Role: LEADER
I20260806 12:27:54.073971 26370 consensus_queue.cc:237] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84db2ce2c6154ec088190c1dce356b0f" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41677 } } peers { permanent_uuid: "a682b8cf6651478dac165186e061b5c7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42463 } } peers { permanent_uuid: "895956dd4ec4482b8edbd98fcf82a35d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44411 } }
I20260806 12:27:54.077821 25634 catalog_manager.cc:5495] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a682b8cf6651478dac165186e061b5c7 (127.28.210.66). New cstate: current_term: 1 leader_uuid: "a682b8cf6651478dac165186e061b5c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84db2ce2c6154ec088190c1dce356b0f" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a682b8cf6651478dac165186e061b5c7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 42463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "895956dd4ec4482b8edbd98fcf82a35d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44411 } health_report { overall_health: UNKNOWN } } }
I20260806 12:27:54.094285 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:54.090269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.28.210.66:42463)
0806 12:27:54.090416 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:54.090421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:54.090436 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:27:54.090573 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:54.090576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:54.090640 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:54.090737 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:27:54.090986 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:54.091486 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:54.092488 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:54.092725 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:54.092969 (+ 244us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:54.093334 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:27:54.093807 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:27:54.093949 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:54.094092 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 12:27:54.094137 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 12:27:54.098196 26332 tablet.cc:2369] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:27:54.102672 25859 raft_consensus.cc:1261] T 60ad38ba318e4d32970016505d29a6e7 P 895956dd4ec4482b8edbd98fcf82a35d [term 1 FOLLOWER]: Refusing update from remote peer a682b8cf6651478dac165186e061b5c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:54.102672 26263 raft_consensus.cc:1261] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f [term 1 FOLLOWER]: Refusing update from remote peer a682b8cf6651478dac165186e061b5c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:54.103518 26378 consensus_queue.cc:1035] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895956dd4ec4482b8edbd98fcf82a35d" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:54.103840 26370 consensus_queue.cc:1035] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84db2ce2c6154ec088190c1dce356b0f" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:54.118198 26388 mvcc.cc:205] Tried to move back new op lower bound from 7315534946715414528 to 7315534946610110464. Current Snapshot: MvccSnapshot[applied={T|T < 7315534946715414528 or (T in {7315534946715414528})}]
I20260806 12:27:54.125214 25634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:54.125442 25634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:54.128614 25634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d41be0fd134a4396b9551965b9c928a9: Sending DeleteTablet for 3 replicas of tablet 60ad38ba318e4d32970016505d29a6e7
I20260806 12:27:54.129621 26232 tablet_service.cc:1514] Processing DeleteTablet for tablet 60ad38ba318e4d32970016505d29a6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:54 UTC) from {username='slave'} at 127.0.0.1:54266
I20260806 12:27:54.130034 26032 tablet_service.cc:1514] Processing DeleteTablet for tablet 60ad38ba318e4d32970016505d29a6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:54 UTC) from {username='slave'} at 127.0.0.1:55926
I20260806 12:27:54.130255 26400 tablet_replica.cc:330] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f: stopping tablet replica
I20260806 12:27:54.130535 25829 tablet_service.cc:1514] Processing DeleteTablet for tablet 60ad38ba318e4d32970016505d29a6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:54 UTC) from {username='slave'} at 127.0.0.1:46060
I20260806 12:27:54.130635 26400 raft_consensus.cc:2229] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:27:54.131023 26400 raft_consensus.cc:2258] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:54.132227 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 25735
I20260806 12:27:54.137637 26400 ts_tablet_manager.cc:1905] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:54.141779 26400 ts_tablet_manager.cc:1918] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:54.142066 26400 log.cc:1196] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/60ad38ba318e4d32970016505d29a6e7
I20260806 12:27:54.142498 26400 ts_tablet_manager.cc:1939] T 60ad38ba318e4d32970016505d29a6e7 P 84db2ce2c6154ec088190c1dce356b0f: Deleting consensus metadata
I20260806 12:27:54.148753 25608 catalog_manager.cc:4841] TS 84db2ce2c6154ec088190c1dce356b0f (127.28.210.67:41677): tablet 60ad38ba318e4d32970016505d29a6e7 (table test-table [id=ed1b10adaee542e594c6a1f437c2c22c]) successfully deleted
I20260806 12:27:54.152468 26401 tablet_replica.cc:330] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7: stopping tablet replica
I20260806 12:27:54.153139 26401 raft_consensus.cc:2229] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:27:54.153868 26401 raft_consensus.cc:2258] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:54.155943 26401 ts_tablet_manager.cc:1905] T 60ad38ba318e4d32970016505d29a6e7 P a682b8cf6651478dac165186e061b5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 12:27:54.156642 25608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:44411 (error 108)
I20260806 12:27:54.156852 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 25936
I20260806 12:27:54.166344 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:54.164693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:44411 (local address 127.0.0.1:46068)
0806 12:27:54.166111 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:54.166230 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1249}
W20260806 12:27:54.166687 25608 catalog_manager.cc:4584] TS 895956dd4ec4482b8edbd98fcf82a35d (127.28.210.65:44411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60ad38ba318e4d32970016505d29a6e7: Network error: Client connection negotiation failed: client connection to 127.28.210.65:44411: connect: Connection refused (error 111)
I20260806 12:27:54.187446 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 26137
I20260806 12:27:54.199247 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:54.187697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:42463 (local address 127.0.0.1:55944)
0806 12:27:54.199005 (+ 11308us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:54.199122 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.66:42463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11135}
I20260806 12:27:54.220999 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:54.220579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:44411 (local address 127.0.0.1:46080)
0806 12:27:54.220790 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:54.220875 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 12:27:54.221832 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:54.221584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:44411 (local address 127.0.0.1:46084)
0806 12:27:54.221694 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:54.221755 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 12:27:54.226504 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 25585
2026-08-06T12:27:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1344 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:27:54 dist-test-slave-lgr5 krb5kdc[26410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:27:54 dist-test-slave-lgr5 krb5kdc[26410](info): set up 2 sockets
Aug 06 12:27:54 dist-test-slave-lgr5 krb5kdc[26410](info): commencing operation
krb5kdc: starting...
W20260806 12:27:55.434512 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:27:55 dist-test-slave-lgr5 krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:27:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:27:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:27:55.581049 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:42515
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:42515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:55.710990 26426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:55.711361 26426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:55.711426 26426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:55.716238 26426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:27:55.716357 26426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:55.716391 26426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:55.716413 26426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:27:55.716435 26426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:27:55.722762 26426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:42515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:42515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:55.724102 26426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:55.725265 26426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:55.734485 26433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:55.734485 26436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:55.734519 26432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:55.734903 26426 server_base.cc:1034] running on GCE node
I20260806 12:27:55.735638 26426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:55.736955 26426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:55.738162 26426 hybrid_clock.cc:648] HybridClock initialized: now 1786019275738143 us; error 34 us; skew 500 ppm
Aug 06 12:27:55 dist-test-slave-lgr5 krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019275, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:55.741474 26426 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:27:55.742563 26426 webserver.cc:460] Webserver started at http://127.28.210.126:35031/ using document root <none> and password file <none>
I20260806 12:27:55.743225 26426 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:55.743307 26426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:55.743530 26426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:55.746001 26426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "ce08bd43a30b49979d028c1011bfdd8f"
format_stamp: "Formatted at 2026-08-06 12:27:55 on dist-test-slave-lgr5"
server_key: "0850e4f278d4c7483f494ee4e58eb2eb"
server_key_iv: "06af1a18d90ac354d6d1bdbf146d685f"
server_key_version: "encryptionkey@0"
I20260806 12:27:55.746677 26426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "ce08bd43a30b49979d028c1011bfdd8f"
format_stamp: "Formatted at 2026-08-06 12:27:55 on dist-test-slave-lgr5"
server_key: "0850e4f278d4c7483f494ee4e58eb2eb"
server_key_iv: "06af1a18d90ac354d6d1bdbf146d685f"
server_key_version: "encryptionkey@0"
I20260806 12:27:55.751461 26426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 12:27:55.754503 26447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:55.755966 26426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:27:55.756150 26426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "ce08bd43a30b49979d028c1011bfdd8f"
format_stamp: "Formatted at 2026-08-06 12:27:55 on dist-test-slave-lgr5"
server_key: "0850e4f278d4c7483f494ee4e58eb2eb"
server_key_iv: "06af1a18d90ac354d6d1bdbf146d685f"
server_key_version: "encryptionkey@0"
I20260806 12:27:55.756282 26426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:55.767683 26426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:55.771071 26426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:55.771279 26426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:55.779204 26426 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:42515
I20260806 12:27:55.779264 26524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:42515 every 8 connection(s)
I20260806 12:27:55.780298 26426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:27:55.783452 26525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:55.790009 26525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f: Bootstrap starting.
I20260806 12:27:55.790437 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 26426
I20260806 12:27:55.790583 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:27:55.790903 29513 external_mini_cluster.cc:1428] Setting key 227aced852feed62156364cecfa498c1
I20260806 12:27:55.793033 26525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:55.794070 26525 log.cc:824] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:55.796197 26525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f: No bootstrap required, opened a new log
I20260806 12:27:55.799942 26525 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce08bd43a30b49979d028c1011bfdd8f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42515 } }
I20260806 12:27:55.800240 26525 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 06 12:27:55 dist-test-slave-lgr5 krb5kdc[26410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:27:55.800297 26525 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce08bd43a30b49979d028c1011bfdd8f, State: Initialized, Role: FOLLOWER
I20260806 12:27:55.800773 26525 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce08bd43a30b49979d028c1011bfdd8f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42515 } }
I20260806 12:27:55.801034 26525 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:27:55.801136 26525 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:27:55.801246 26525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:55.802376 26525 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce08bd43a30b49979d028c1011bfdd8f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42515 } }
I20260806 12:27:55.802728 26525 leader_election.cc:302] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce08bd43a30b49979d028c1011bfdd8f; no voters:
I20260806 12:27:55.803064 26525 leader_election.cc:288] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:27:55.803195 26532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:55.803442 26532 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [term 1 LEADER]: Becoming Leader. State: Replica: ce08bd43a30b49979d028c1011bfdd8f, State: Running, Role: LEADER
I20260806 12:27:55.803771 26532 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce08bd43a30b49979d028c1011bfdd8f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42515 } }
I20260806 12:27:55.804587 26525 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:27:55.805668 26533 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce08bd43a30b49979d028c1011bfdd8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce08bd43a30b49979d028c1011bfdd8f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42515 } } }
I20260806 12:27:55.805796 26533 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:55.806121 26534 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce08bd43a30b49979d028c1011bfdd8f. Latest consensus state: current_term: 1 leader_uuid: "ce08bd43a30b49979d028c1011bfdd8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce08bd43a30b49979d028c1011bfdd8f" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 42515 } } }
I20260806 12:27:55.806208 26534 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:55.806519 26545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:27:55.806664 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:55.792585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.28.210.126:42515)
0806 12:27:55.793045 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:55.793061 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:55.793108 (+ 47us) server_negotiation.cc:407] Connection header received
0806 12:27:55.794196 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:55.794225 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:55.794603 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:55.794929 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:55.795782 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:55.796879 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:55.797752 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:55.798241 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:55.798392 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:55.800861 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:55.800867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:55.800883 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:55.800916 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:55.802691 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:55.803240 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:55.803247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:55.803253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:55.803412 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:55.803764 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:55.803767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:55.803769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:55.804341 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:55.804550 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:55.805236 (+ 686us) server_negotiation.cc:299] Negotiation successful
0806 12:27:55.805584 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":144,"threads_started":1}
I20260806 12:27:55.810113 26545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:27:55.817034 26545 catalog_manager.cc:1347] Generated new cluster ID: 1182e493eeb74f0fbb2e9ef49e825dd3
I20260806 12:27:55.817121 26545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:27:55.844489 26545 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:27:55.845546 26545 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:27:55.850653 26545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f: Generated new TSK 0
I20260806 12:27:55.851459 26545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
Entry for principal kudu/127.28.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab.
I20260806 12:27:55.898939 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42515
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:56.039036 26562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:56.039376 26562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:56.039424 26562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:56.043853 26562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:56.043964 26562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:56.044241 26562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:27:56.050705 26562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.65.keytab
--principal=kudu/127.28.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:56.051994 26562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:56.053194 26562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:56.068605 26569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:56.068593 26575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:56.068609 26570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:56.068677 26562 server_base.cc:1034] running on GCE node
I20260806 12:27:56.069339 26562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:56.070175 26562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:56.071411 26562 hybrid_clock.cc:648] HybridClock initialized: now 1786019276071386 us; error 37 us; skew 500 ppm
Aug 06 12:27:56 dist-test-slave-lgr5 krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:56.074771 26562 init.cc:377] Logged in from keytab as kudu/127.28.210.65@KRBTEST.COM (short username kudu)
I20260806 12:27:56.075912 26562 webserver.cc:460] Webserver started at http://127.28.210.65:43415/ using document root <none> and password file <none>
I20260806 12:27:56.076603 26562 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:56.076678 26562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:56.076875 26562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:56.079591 26562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "5296d63ae4f441da8d4cf1a9d23382c5"
format_stamp: "Formatted at 2026-08-06 12:27:56 on dist-test-slave-lgr5"
server_key: "5b9230fd58a5c137df1c9416c893cbf9"
server_key_iv: "5850840e595d7d932f2baf64f6d7e772"
server_key_version: "encryptionkey@0"
I20260806 12:27:56.080425 26562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "5296d63ae4f441da8d4cf1a9d23382c5"
format_stamp: "Formatted at 2026-08-06 12:27:56 on dist-test-slave-lgr5"
server_key: "5b9230fd58a5c137df1c9416c893cbf9"
server_key_iv: "5850840e595d7d932f2baf64f6d7e772"
server_key_version: "encryptionkey@0"
I20260806 12:27:56.085594 26562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 12:27:56.088874 26586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:56.090569 26562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:27:56.090715 26562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "5296d63ae4f441da8d4cf1a9d23382c5"
format_stamp: "Formatted at 2026-08-06 12:27:56 on dist-test-slave-lgr5"
server_key: "5b9230fd58a5c137df1c9416c893cbf9"
server_key_iv: "5850840e595d7d932f2baf64f6d7e772"
server_key_version: "encryptionkey@0"
I20260806 12:27:56.090842 26562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:56.115548 26562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:56.119424 26562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:56.119611 26562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:56.120263 26562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:56.121227 26562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:56.121291 26562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:56.121361 26562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:56.121387 26562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:56.131633 26562 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:44427
I20260806 12:27:56.131668 26752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:44427 every 8 connection(s)
I20260806 12:27:56.132802 26562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:27:56.139062 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 26562
I20260806 12:27:56.139230 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:27:56.139566 29513 external_mini_cluster.cc:1428] Setting key 71b81ad7728feb1df536be3ce2b9e1d3
Aug 06 12:27:56 dist-test-slave-lgr5 krb5kdc[26410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.65@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:27:56.150879 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.135114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:56603 (local address 127.28.210.126:42515)
0806 12:27:56.135267 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:56.135272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:56.136366 (+ 1094us) server_negotiation.cc:407] Connection header received
0806 12:27:56.137908 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:56.137915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:56.137993 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:56.138078 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:56.140068 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.140732 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.142484 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.142846 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.142955 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.146463 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:56.146467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:56.146469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:56.146501 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:56.148440 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:56.149079 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:56.149082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:56.149083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:56.149171 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:56.149576 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:56.149579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:56.149580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:56.149793 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:56.149886 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:56.150498 (+ 612us) server_negotiation.cc:299] Negotiation successful
0806 12:27:56.150680 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.28.210.66@KRBTEST.COM; defaulting to no policy
I20260806 12:27:56.151940 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.135542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:42515 (local address 127.28.210.65:56603)
0806 12:27:56.136190 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:56.136244 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:56.137546 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:56.138420 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:56.138431 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:56.138974 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:56.139832 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.139859 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.140891 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.140895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.142222 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.142240 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.143065 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.143071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.143309 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.144322 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:56.144342 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:56.146214 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:56.148596 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:56.148604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:56.148625 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:56.148963 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:56.149265 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:56.149268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:56.149271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:56.149468 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:56.149912 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:56.149920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:56.150233 (+ 313us) client_negotiation.cc:769] Sending connection context
0806 12:27:56.150434 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 12:27:56.150874 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":155,"threads_started":1}
I20260806 12:27:56.154078 26754 heartbeater.cc:347] Connected to a master server at 127.28.210.126:42515
I20260806 12:27:56.154453 26754 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:56.155328 26754 heartbeater.cc:511] Master 127.28.210.126:42515 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.66@KRBTEST.COM" created.
I20260806 12:27:56.157889 26469 ts_manager.cc:194] Registered new tserver with Master: 5296d63ae4f441da8d4cf1a9d23382c5 (127.28.210.65:44427)
I20260806 12:27:56.159607 26469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.65@KRBTEST.COM'} at 127.28.210.65:56603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
Entry for principal kudu/127.28.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab.
I20260806 12:27:56.175810 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42515
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:56.312120 26766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:56.312557 26766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:56.312609 26766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:56.317582 26766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:56.317704 26766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:56.317780 26766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:27:56.324898 26766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.66.keytab
--principal=kudu/127.28.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:56.326323 26766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:56.327560 26766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:56.339893 26782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:56.340448 26779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:56.340509 26780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:56.340790 26766 server_base.cc:1034] running on GCE node
I20260806 12:27:56.341300 26766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:56.342180 26766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:56.343634 26766 hybrid_clock.cc:648] HybridClock initialized: now 1786019276343446 us; error 45 us; skew 500 ppm
Aug 06 12:27:56 dist-test-slave-lgr5 krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:56.347216 26766 init.cc:377] Logged in from keytab as kudu/127.28.210.66@KRBTEST.COM (short username kudu)
I20260806 12:27:56.348654 26766 webserver.cc:460] Webserver started at http://127.28.210.66:38657/ using document root <none> and password file <none>
I20260806 12:27:56.349334 26766 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:56.349418 26766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:56.349628 26766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:56.352248 26766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "f941b7a9faeb42da9279d42c3bc189e7"
format_stamp: "Formatted at 2026-08-06 12:27:56 on dist-test-slave-lgr5"
server_key: "3e04dce0340f148ab409c41f7fb47e1a"
server_key_iv: "aad8b5857c9a085c371fd7fb9eca1ffd"
server_key_version: "encryptionkey@0"
I20260806 12:27:56.352924 26766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "f941b7a9faeb42da9279d42c3bc189e7"
format_stamp: "Formatted at 2026-08-06 12:27:56 on dist-test-slave-lgr5"
server_key: "3e04dce0340f148ab409c41f7fb47e1a"
server_key_iv: "aad8b5857c9a085c371fd7fb9eca1ffd"
server_key_version: "encryptionkey@0"
I20260806 12:27:56.357806 26766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 12:27:56.360993 26793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:56.362696 26766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:27:56.362859 26766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "f941b7a9faeb42da9279d42c3bc189e7"
format_stamp: "Formatted at 2026-08-06 12:27:56 on dist-test-slave-lgr5"
server_key: "3e04dce0340f148ab409c41f7fb47e1a"
server_key_iv: "aad8b5857c9a085c371fd7fb9eca1ffd"
server_key_version: "encryptionkey@0"
I20260806 12:27:56.363003 26766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:56.381198 26766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:56.385380 26766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:56.385625 26766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:56.386379 26766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:56.387425 26766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:56.387485 26766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:56.387545 26766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:56.387578 26766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:56.397787 26766 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:36815
I20260806 12:27:56.397954 26956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:36815 every 8 connection(s)
I20260806 12:27:56.398885 26766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:27:56.405527 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 26766
I20260806 12:27:56.405699 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:27:56.406064 29513 external_mini_cluster.cc:1428] Setting key 142ef6ca1e253ea09e23ee35559e5430
Aug 06 12:27:56 dist-test-slave-lgr5 krb5kdc[26410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.66@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:27:56.417508 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.401329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:43575 (local address 127.28.210.126:42515)
0806 12:27:56.401534 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:56.401538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:56.402633 (+ 1095us) server_negotiation.cc:407] Connection header received
0806 12:27:56.404288 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:56.404292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:56.404382 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:56.404463 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:56.406466 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.407172 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.408806 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.409136 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.409254 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.412656 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:56.412660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:56.412663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:56.412695 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:56.414421 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:56.415323 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:56.415327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:56.415328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:56.415432 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:56.415914 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:56.415916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:56.415918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:56.416207 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:56.416297 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:56.417211 (+ 914us) server_negotiation.cc:299] Negotiation successful
0806 12:27:56.417359 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 12:27:56.418843 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.401759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:42515 (local address 127.28.210.66:43575)
0806 12:27:56.402416 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:56.402474 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:56.403950 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:56.404718 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:56.404731 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:56.405292 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:56.406204 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.406233 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.407365 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.407369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.408647 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.408663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.409334 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.409338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.409529 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.410256 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:56.410275 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:56.412376 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:56.414611 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:56.414620 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:56.414646 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:56.415145 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:56.415544 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:56.415547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:56.415549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:56.415798 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:56.416334 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:56.416343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:56.416790 (+ 447us) client_negotiation.cc:769] Sending connection context
0806 12:27:56.417058 (+ 268us) client_negotiation.cc:240] Negotiation successful
0806 12:27:56.417528 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":150,"threads_started":1}
WARNING: no policy specified for kudu/127.28.210.67@KRBTEST.COM; defaulting to no policy
I20260806 12:27:56.420616 26957 heartbeater.cc:347] Connected to a master server at 127.28.210.126:42515
I20260806 12:27:56.421058 26957 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:56.421785 26957 heartbeater.cc:511] Master 127.28.210.126:42515 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.67@KRBTEST.COM" created.
I20260806 12:27:56.423347 26469 ts_manager.cc:194] Registered new tserver with Master: f941b7a9faeb42da9279d42c3bc189e7 (127.28.210.66:36815)
I20260806 12:27:56.424222 26469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.66@KRBTEST.COM'} at 127.28.210.66:43575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
Entry for principal kudu/127.28.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab.
I20260806 12:27:56.445763 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42515
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:56.585814 26964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:56.586191 26964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:56.586242 26964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:56.591084 26964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:56.591188 26964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:56.591252 26964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:27:56.597391 26964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.67.keytab
--principal=kudu/127.28.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:42515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:56.598703 26964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:56.599850 26964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:56.612286 26981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:56.612299 26983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:56.612303 26975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:56.612360 26964 server_base.cc:1034] running on GCE node
I20260806 12:27:56.613248 26964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:56.614202 26964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:56.615417 26964 hybrid_clock.cc:648] HybridClock initialized: now 1786019276615366 us; error 75 us; skew 500 ppm
Aug 06 12:27:56 dist-test-slave-lgr5 krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:56.623730 26964 init.cc:377] Logged in from keytab as kudu/127.28.210.67@KRBTEST.COM (short username kudu)
I20260806 12:27:56.625101 26964 webserver.cc:460] Webserver started at http://127.28.210.67:33617/ using document root <none> and password file <none>
I20260806 12:27:56.625784 26964 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:56.625869 26964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:56.626104 26964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:56.628597 26964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "54291912af784569b7b828aee21234bf"
format_stamp: "Formatted at 2026-08-06 12:27:56 on dist-test-slave-lgr5"
server_key: "8695d0abcbd584e56681f7c025d9c3f4"
server_key_iv: "2df4f1c8f0b9fe77ea2fca0ce94472e0"
server_key_version: "encryptionkey@0"
I20260806 12:27:56.629247 26964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "54291912af784569b7b828aee21234bf"
format_stamp: "Formatted at 2026-08-06 12:27:56 on dist-test-slave-lgr5"
server_key: "8695d0abcbd584e56681f7c025d9c3f4"
server_key_iv: "2df4f1c8f0b9fe77ea2fca0ce94472e0"
server_key_version: "encryptionkey@0"
I20260806 12:27:56.633744 26964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:27:56.636672 26993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:56.638370 26964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:56.638569 26964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "54291912af784569b7b828aee21234bf"
format_stamp: "Formatted at 2026-08-06 12:27:56 on dist-test-slave-lgr5"
server_key: "8695d0abcbd584e56681f7c025d9c3f4"
server_key_iv: "2df4f1c8f0b9fe77ea2fca0ce94472e0"
server_key_version: "encryptionkey@0"
I20260806 12:27:56.638710 26964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:56.667511 26964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:56.671913 26964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:56.672284 26964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:56.673060 26964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:56.674367 26964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:56.674429 26964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:56.674495 26964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:56.674528 26964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:56.686534 26964 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:42947
I20260806 12:27:56.686555 27177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:42947 every 8 connection(s)
I20260806 12:27:56.687837 26964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:27:56.696857 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 26964
I20260806 12:27:56.697031 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:27:56.697413 29513 external_mini_cluster.cc:1428] Setting key acbffa81e1ffaecf4cabddea0ff3e9de
Aug 06 12:27:56 dist-test-slave-lgr5 krb5kdc[26410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.67@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:27:56.705600 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.690262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:37389 (local address 127.28.210.126:42515)
0806 12:27:56.690489 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:56.690494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:56.691524 (+ 1030us) server_negotiation.cc:407] Connection header received
0806 12:27:56.693257 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:56.693261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:56.693334 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:56.693415 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:56.695153 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.695787 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.697710 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.697967 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.698097 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.701549 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:56.701552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:56.701554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:56.701586 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:56.703009 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:56.703680 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:56.703684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:56.703685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:56.703783 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:56.704296 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:56.704300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:56.704301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:56.704575 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:56.704669 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:56.705298 (+ 629us) server_negotiation.cc:299] Negotiation successful
0806 12:27:56.705453 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 12:27:56.706631 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.690694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:42515 (local address 127.28.210.67:37389)
0806 12:27:56.691316 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:56.691375 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:56.692897 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:56.693665 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:56.693676 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:56.694195 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:56.694870 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.694896 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.695973 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.695976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.697515 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.697533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.698160 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.698163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.698431 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.699335 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:27:56.699359 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:56.701318 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:56.703157 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:56.703164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:56.703182 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:56.703498 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:56.703927 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:27:56.703930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:27:56.703931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:27:56.704177 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:27:56.704674 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:56.704681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:56.705008 (+ 327us) client_negotiation.cc:769] Sending connection context
0806 12:27:56.705194 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 12:27:56.705579 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":157,"threads_started":1}
I20260806 12:27:56.708032 27178 heartbeater.cc:347] Connected to a master server at 127.28.210.126:42515
I20260806 12:27:56.708341 27178 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:56.708942 27178 heartbeater.cc:511] Master 127.28.210.126:42515 requested a full tablet report, sending...
I20260806 12:27:56.710184 26469 ts_manager.cc:194] Registered new tserver with Master: 54291912af784569b7b828aee21234bf (127.28.210.67:42947)
I20260806 12:27:56.710830 26469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.210.67@KRBTEST.COM'} at 127.28.210.67:37389
I20260806 12:27:56.712473 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:27:56.723492 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.714852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56634 (local address 127.28.210.126:42515)
0806 12:27:56.715071 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:56.715077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:56.715199 (+ 122us) server_negotiation.cc:407] Connection header received
0806 12:27:56.715348 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:56.715351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:56.715427 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:56.715498 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:56.716381 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.716980 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.717872 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.718123 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.718227 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.719792 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:56.719796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:56.719799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:56.719836 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:56.721534 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:56.722059 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:56.722063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:56.722064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:56.722160 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:56.722437 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:56.722441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:56.722443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:56.722711 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:56.722845 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:56.723142 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 12:27:56.723277 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 12:27:56.727058 26469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:27:56.730705 26469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:27:56.747825 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.741563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:44427 (local address 127.0.0.1:40026)
0806 12:27:56.742345 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:56.742359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:56.742552 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:56.743313 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:56.743316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:56.743386 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:56.743777 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.743783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.744885 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.744889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.746145 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.746157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.747204 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.747207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.747321 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.747376 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 12:27:56.747495 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 12:27:56.747657 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":273,"spinlock_wait_cycles":16000,"thread_start_us":102,"threads_started":1}
I20260806 12:27:56.747835 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.742158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.28.210.65:44427)
0806 12:27:56.742798 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:56.742805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:56.742830 (+ 25us) server_negotiation.cc:407] Connection header received
0806 12:27:56.742916 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:56.742922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:56.743146 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:56.743476 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:56.743900 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.744761 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.746279 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.747076 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.747181 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.747358 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:56.747550 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 12:27:56.747657 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":83,"threads_started":1}
I20260806 12:27:56.748242 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.742048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:36815 (local address 127.0.0.1:59078)
0806 12:27:56.742382 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:56.742395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:56.742483 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:56.743313 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:56.743316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:56.743387 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:56.743572 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.743581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.745300 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.745304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.746607 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.746619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.747864 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.747868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.747945 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.747965 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 12:27:56.748029 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 12:27:56.748125 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":140,"spinlock_wait_cycles":58368,"thread_start_us":86,"threads_started":1}
I20260806 12:27:56.748070 27193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.741206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:42947 (local address 127.0.0.1:54096)
0806 12:27:56.741892 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:56.741950 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:56.742118 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:56.743309 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:56.743315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:56.743341 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:56.743572 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.743581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.745300 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.745304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.746500 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.746512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.747661 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.747665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.747756 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.747779 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 12:27:56.747833 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 12:27:56.747912 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":33,"spinlock_wait_cycles":1408,"thread_start_us":115,"threads_started":1}
I20260806 12:27:56.748698 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.741531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.28.210.67:42947)
0806 12:27:56.742745 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:56.742758 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:56.742784 (+ 26us) server_negotiation.cc:407] Connection header received
0806 12:27:56.742868 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:56.742874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:56.743134 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:56.743295 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:56.743742 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.745124 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.746631 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.747504 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.748210 (+ 706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.748385 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:56.748446 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 12:27:56.748547 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":140,"threads_started":1}
I20260806 12:27:56.748698 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.742302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.28.210.66:36815)
0806 12:27:56.742796 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:56.742807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:56.742830 (+ 23us) server_negotiation.cc:407] Connection header received
0806 12:27:56.742958 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:56.742973 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:56.743145 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:56.743492 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:56.743734 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.745111 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.746744 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.747541 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.747639 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.747788 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:56.748469 (+ 681us) server_negotiation.cc:299] Negotiation successful
0806 12:27:56.748567 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":127,"threads_started":1}
I20260806 12:27:56.751814 26862 tablet_service.cc:1467] Processing CreateTablet for tablet 7b01d86c9ffa4ce7a644e42fe70fcd3c (DEFAULT_TABLE table=test-table [id=41c65fe9bc214c168aa9b0d73224e030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:56.751839 27071 tablet_service.cc:1467] Processing CreateTablet for tablet 7b01d86c9ffa4ce7a644e42fe70fcd3c (DEFAULT_TABLE table=test-table [id=41c65fe9bc214c168aa9b0d73224e030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:56.751839 26655 tablet_service.cc:1467] Processing CreateTablet for tablet 7b01d86c9ffa4ce7a644e42fe70fcd3c (DEFAULT_TABLE table=test-table [id=41c65fe9bc214c168aa9b0d73224e030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:56.753221 26655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b01d86c9ffa4ce7a644e42fe70fcd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:56.753222 26862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b01d86c9ffa4ce7a644e42fe70fcd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:56.753571 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b01d86c9ffa4ce7a644e42fe70fcd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:56.762535 27204 tablet_bootstrap.cc:492] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7: Bootstrap starting.
I20260806 12:27:56.763418 27205 tablet_bootstrap.cc:492] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5: Bootstrap starting.
I20260806 12:27:56.764727 27206 tablet_bootstrap.cc:492] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf: Bootstrap starting.
I20260806 12:27:56.765054 27204 tablet_bootstrap.cc:654] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:56.765631 27205 tablet_bootstrap.cc:654] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:56.766391 27204 log.cc:824] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:56.766536 27205 log.cc:824] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:56.766989 27206 tablet_bootstrap.cc:654] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:56.768484 27206 log.cc:824] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:56.769332 27204 tablet_bootstrap.cc:492] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7: No bootstrap required, opened a new log
I20260806 12:27:56.769708 27204 ts_tablet_manager.cc:1397] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 12:27:56.771173 27206 tablet_bootstrap.cc:492] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf: No bootstrap required, opened a new log
I20260806 12:27:56.771456 27206 ts_tablet_manager.cc:1397] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 12:27:56.773332 27205 tablet_bootstrap.cc:492] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5: No bootstrap required, opened a new log
I20260806 12:27:56.773639 27205 ts_tablet_manager.cc:1397] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20260806 12:27:56.774511 27204 raft_consensus.cc:348] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54291912af784569b7b828aee21234bf" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 42947 } } peers { permanent_uuid: "5296d63ae4f441da8d4cf1a9d23382c5" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44427 } } peers { permanent_uuid: "f941b7a9faeb42da9279d42c3bc189e7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36815 } }
I20260806 12:27:56.774896 27204 raft_consensus.cc:374] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:56.774968 27204 raft_consensus.cc:729] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f941b7a9faeb42da9279d42c3bc189e7, State: Initialized, Role: FOLLOWER
I20260806 12:27:56.775655 27204 consensus_queue.cc:260] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54291912af784569b7b828aee21234bf" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 42947 } } peers { permanent_uuid: "5296d63ae4f441da8d4cf1a9d23382c5" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44427 } } peers { permanent_uuid: "f941b7a9faeb42da9279d42c3bc189e7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36815 } }
I20260806 12:27:56.776126 27206 raft_consensus.cc:348] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54291912af784569b7b828aee21234bf" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 42947 } } peers { permanent_uuid: "5296d63ae4f441da8d4cf1a9d23382c5" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44427 } } peers { permanent_uuid: "f941b7a9faeb42da9279d42c3bc189e7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36815 } }
I20260806 12:27:56.776504 27206 raft_consensus.cc:374] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:56.776583 27206 raft_consensus.cc:729] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54291912af784569b7b828aee21234bf, State: Initialized, Role: FOLLOWER
I20260806 12:27:56.776815 26957 heartbeater.cc:503] Master 127.28.210.126:42515 was elected leader, sending a full tablet report...
I20260806 12:27:56.777181 27204 ts_tablet_manager.cc:1428] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260806 12:27:56.777175 27206 consensus_queue.cc:260] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54291912af784569b7b828aee21234bf" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 42947 } } peers { permanent_uuid: "5296d63ae4f441da8d4cf1a9d23382c5" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44427 } } peers { permanent_uuid: "f941b7a9faeb42da9279d42c3bc189e7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36815 } }
I20260806 12:27:56.778023 27178 heartbeater.cc:503] Master 127.28.210.126:42515 was elected leader, sending a full tablet report...
I20260806 12:27:56.778403 27206 ts_tablet_manager.cc:1428] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 12:27:56.778692 27205 raft_consensus.cc:348] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54291912af784569b7b828aee21234bf" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 42947 } } peers { permanent_uuid: "5296d63ae4f441da8d4cf1a9d23382c5" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44427 } } peers { permanent_uuid: "f941b7a9faeb42da9279d42c3bc189e7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36815 } }
I20260806 12:27:56.779134 27205 raft_consensus.cc:374] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:56.779193 27205 raft_consensus.cc:729] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5296d63ae4f441da8d4cf1a9d23382c5, State: Initialized, Role: FOLLOWER
I20260806 12:27:56.779696 27205 consensus_queue.cc:260] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54291912af784569b7b828aee21234bf" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 42947 } } peers { permanent_uuid: "5296d63ae4f441da8d4cf1a9d23382c5" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44427 } } peers { permanent_uuid: "f941b7a9faeb42da9279d42c3bc189e7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36815 } }
I20260806 12:27:56.780774 27205 ts_tablet_manager.cc:1428] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 12:27:56.781447 26754 heartbeater.cc:503] Master 127.28.210.126:42515 was elected leader, sending a full tablet report...
I20260806 12:27:56.851025 27211 raft_consensus.cc:482] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:27:56.851280 27211 raft_consensus.cc:504] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54291912af784569b7b828aee21234bf" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 42947 } } peers { permanent_uuid: "5296d63ae4f441da8d4cf1a9d23382c5" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44427 } } peers { permanent_uuid: "f941b7a9faeb42da9279d42c3bc189e7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36815 } }
I20260806 12:27:56.852806 27211 leader_election.cc:288] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54291912af784569b7b828aee21234bf (127.28.210.67:42947), 5296d63ae4f441da8d4cf1a9d23382c5 (127.28.210.65:44427)
I20260806 12:27:56.857862 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.853460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:41633 (local address 127.28.210.67:42947)
0806 12:27:56.853603 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:56.853608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:56.853753 (+ 145us) server_negotiation.cc:407] Connection header received
0806 12:27:56.853844 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:56.853846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:56.853895 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:56.853969 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:56.854207 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.854985 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.856857 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.857277 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.857412 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.857477 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:56.857651 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 12:27:56.857722 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 12:27:56.857930 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.853187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:42947 (local address 127.28.210.66:41633)
0806 12:27:56.853663 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:56.853675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:56.853762 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:56.853988 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:56.853991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:56.854005 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:56.854107 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.854113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.855161 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.855166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.856668 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.856684 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.857413 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.857417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.857521 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.857546 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 12:27:56.857616 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 12:27:56.857701 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":84,"thread_start_us":149,"threads_started":1}
I20260806 12:27:56.857960 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.853402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:40783 (local address 127.28.210.65:44427)
0806 12:27:56.853613 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:56.853618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:56.853634 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:27:56.853699 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:56.853701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:56.853755 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:56.853848 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:56.854158 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.854985 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.856853 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.857308 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.857448 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.857509 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:56.857717 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 12:27:56.857798 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260806 12:27:56.857936 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.853170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:44427 (local address 127.28.210.66:40783)
0806 12:27:56.853404 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:56.853420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:56.853537 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:56.853882 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:56.853885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:56.853901 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:56.854034 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.854041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.855161 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.855166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.856668 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:56.856684 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.857478 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.857482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:56.857596 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.857616 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 12:27:56.857689 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 12:27:56.857770 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":92}
I20260806 12:27:56.858796 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b01d86c9ffa4ce7a644e42fe70fcd3c" candidate_uuid: "f941b7a9faeb42da9279d42c3bc189e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5296d63ae4f441da8d4cf1a9d23382c5" is_pre_election: true
I20260806 12:27:56.858801 27100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b01d86c9ffa4ce7a644e42fe70fcd3c" candidate_uuid: "f941b7a9faeb42da9279d42c3bc189e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54291912af784569b7b828aee21234bf" is_pre_election: true
I20260806 12:27:56.859186 27100 raft_consensus.cc:2454] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f941b7a9faeb42da9279d42c3bc189e7 in term 0.
I20260806 12:27:56.859186 26687 raft_consensus.cc:2454] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f941b7a9faeb42da9279d42c3bc189e7 in term 0.
I20260806 12:27:56.859843 26795 leader_election.cc:302] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54291912af784569b7b828aee21234bf, f941b7a9faeb42da9279d42c3bc189e7; no voters:
I20260806 12:27:56.860267 27211 raft_consensus.cc:2781] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:27:56.860520 27211 raft_consensus.cc:482] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:27:56.860595 27211 raft_consensus.cc:3037] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:56.861821 27211 raft_consensus.cc:504] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54291912af784569b7b828aee21234bf" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 42947 } } peers { permanent_uuid: "5296d63ae4f441da8d4cf1a9d23382c5" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44427 } } peers { permanent_uuid: "f941b7a9faeb42da9279d42c3bc189e7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36815 } }
I20260806 12:27:56.862226 27211 leader_election.cc:288] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [CANDIDATE]: Term 1 election: Requested vote from peers 54291912af784569b7b828aee21234bf (127.28.210.67:42947), 5296d63ae4f441da8d4cf1a9d23382c5 (127.28.210.65:44427)
I20260806 12:27:56.862787 27100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b01d86c9ffa4ce7a644e42fe70fcd3c" candidate_uuid: "f941b7a9faeb42da9279d42c3bc189e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54291912af784569b7b828aee21234bf"
I20260806 12:27:56.862787 26687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b01d86c9ffa4ce7a644e42fe70fcd3c" candidate_uuid: "f941b7a9faeb42da9279d42c3bc189e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5296d63ae4f441da8d4cf1a9d23382c5"
I20260806 12:27:56.862944 27100 raft_consensus.cc:3037] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:56.862946 26687 raft_consensus.cc:3037] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:56.864419 27100 raft_consensus.cc:2454] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f941b7a9faeb42da9279d42c3bc189e7 in term 1.
I20260806 12:27:56.864419 26687 raft_consensus.cc:2454] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f941b7a9faeb42da9279d42c3bc189e7 in term 1.
I20260806 12:27:56.864832 26795 leader_election.cc:302] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54291912af784569b7b828aee21234bf, f941b7a9faeb42da9279d42c3bc189e7; no voters:
I20260806 12:27:56.865063 27211 raft_consensus.cc:2781] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:56.865299 27211 raft_consensus.cc:686] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 1 LEADER]: Becoming Leader. State: Replica: f941b7a9faeb42da9279d42c3bc189e7, State: Running, Role: LEADER
I20260806 12:27:56.865738 27211 consensus_queue.cc:237] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54291912af784569b7b828aee21234bf" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 42947 } } peers { permanent_uuid: "5296d63ae4f441da8d4cf1a9d23382c5" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44427 } } peers { permanent_uuid: "f941b7a9faeb42da9279d42c3bc189e7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36815 } }
I20260806 12:27:56.869577 26470 catalog_manager.cc:5495] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f941b7a9faeb42da9279d42c3bc189e7 (127.28.210.66). New cstate: current_term: 1 leader_uuid: "f941b7a9faeb42da9279d42c3bc189e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54291912af784569b7b828aee21234bf" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 42947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5296d63ae4f441da8d4cf1a9d23382c5" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f941b7a9faeb42da9279d42c3bc189e7" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 36815 } health_report { overall_health: HEALTHY } } }
W20260806 12:27:56.884428 26755 tablet.cc:2369] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 12:27:56.900408 26959 tablet.cc:2369] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:27:56.909119 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.905608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59086 (local address 127.28.210.66:36815)
0806 12:27:56.905766 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:56.905771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:56.905789 (+ 18us) server_negotiation.cc:407] Connection header received
0806 12:27:56.905878 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:56.905880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:56.905941 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:56.906033 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:27:56.906272 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.906806 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.907690 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:56.907899 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:56.907968 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:56.908281 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:27:56.908760 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:27:56.908853 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:56.908887 (+ 34us) server_negotiation.cc:299] Negotiation successful
0806 12:27:56.908942 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 12:27:56.915380 27100 raft_consensus.cc:1261] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf [term 1 FOLLOWER]: Refusing update from remote peer f941b7a9faeb42da9279d42c3bc189e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:56.916019 26687 raft_consensus.cc:1261] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 5296d63ae4f441da8d4cf1a9d23382c5 [term 1 FOLLOWER]: Refusing update from remote peer f941b7a9faeb42da9279d42c3bc189e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:56.916172 27211 consensus_queue.cc:1035] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54291912af784569b7b828aee21234bf" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 42947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:56.916680 27218 consensus_queue.cc:1035] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5296d63ae4f441da8d4cf1a9d23382c5" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:56.925742 27229 mvcc.cc:205] Tried to move back new op lower bound from 7315534958237519872 to 7315534958045052928. Current Snapshot: MvccSnapshot[applied={T|T < 7315534958237519872}]
I20260806 12:27:56.935281 26470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:56.935523 26470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:56.938834 26470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce08bd43a30b49979d028c1011bfdd8f: Sending DeleteTablet for 3 replicas of tablet 7b01d86c9ffa4ce7a644e42fe70fcd3c
I20260806 12:27:56.940160 26862 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b01d86c9ffa4ce7a644e42fe70fcd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:56 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:59078
I20260806 12:27:56.940181 27071 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b01d86c9ffa4ce7a644e42fe70fcd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:56 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:54096
I20260806 12:27:56.940820 27243 tablet_replica.cc:330] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf: stopping tablet replica
I20260806 12:27:56.941272 27243 raft_consensus.cc:2229] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:27:56.941220 26655 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b01d86c9ffa4ce7a644e42fe70fcd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:56 UTC) from {username='kudu', principal='kudu/127.28.210.126@KRBTEST.COM'} at 127.0.0.1:40026
I20260806 12:27:56.941676 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 26562
I20260806 12:27:56.944634 27243 raft_consensus.cc:2258] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:56.947820 27243 ts_tablet_manager.cc:1905] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:56.950630 27243 ts_tablet_manager.cc:1918] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:56.950754 27243 log.cc:1196] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/7b01d86c9ffa4ce7a644e42fe70fcd3c
I20260806 12:27:56.951061 27243 ts_tablet_manager.cc:1939] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P 54291912af784569b7b828aee21234bf: Deleting consensus metadata
I20260806 12:27:56.952452 27241 tablet_replica.cc:330] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7: stopping tablet replica
I20260806 12:27:56.952492 26449 catalog_manager.cc:4841] TS 54291912af784569b7b828aee21234bf (127.28.210.67:42947): tablet 7b01d86c9ffa4ce7a644e42fe70fcd3c (table test-table [id=41c65fe9bc214c168aa9b0d73224e030]) successfully deleted
I20260806 12:27:56.952844 27241 raft_consensus.cc:2229] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 12:27:56.953269 27241 raft_consensus.cc:2258] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:56.954978 27241 ts_tablet_manager.cc:1905] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:56.958156 27241 ts_tablet_manager.cc:1918] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:56.958277 27241 log.cc:1196] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/7b01d86c9ffa4ce7a644e42fe70fcd3c
I20260806 12:27:56.958626 27241 ts_tablet_manager.cc:1939] T 7b01d86c9ffa4ce7a644e42fe70fcd3c P f941b7a9faeb42da9279d42c3bc189e7: Deleting consensus metadata
W20260806 12:27:56.959419 26454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:44427 (error 108)
I20260806 12:27:56.959619 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 26766
I20260806 12:27:56.963897 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.963530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:44427 (local address 127.0.0.1:40032)
0806 12:27:56.963691 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:56.963780 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:44427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 12:27:56.968115 26454 catalog_manager.cc:4584] TS 5296d63ae4f441da8d4cf1a9d23382c5 (127.28.210.65:44427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b01d86c9ffa4ce7a644e42fe70fcd3c: Network error: Client connection negotiation failed: client connection to 127.28.210.65:44427: connect: Connection refused (error 111)
I20260806 12:27:56.974922 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 26964
I20260806 12:27:56.986676 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:56.975182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:36815 (local address 127.0.0.1:59102)
0806 12:27:56.986438 (+ 11256us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:56.986548 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.66:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11101}
I20260806 12:27:56.993021 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 26426
2026-08-06T12:27:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2771 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T12:27:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:27:57Z Disabled control of system clock
I20260806 12:27:57.039788 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:36649
--webserver_interface=127.28.210.126
--webserver_port=0
--builtin_ntp_servers=127.28.210.84:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:36649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 12:27:57.165302 27251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:57.165652 27251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:57.165690 27251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:57.170328 27251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:27:57.170433 27251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:57.170449 27251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:57.170466 27251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:27:57.170482 27251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:27:57.176425 27251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:36649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:36649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:57.177772 27251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:57.179322 27251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:57.188863 27263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:57.188932 27258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:57.189016 27251 server_base.cc:1034] running on GCE node
W20260806 12:27:57.188856 27259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:57.190068 27251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:57.191352 27251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:57.192535 27251 hybrid_clock.cc:648] HybridClock initialized: now 1786019277192492 us; error 59 us; skew 500 ppm
I20260806 12:27:57.194900 27251 webserver.cc:460] Webserver started at http://127.28.210.126:39903/ using document root <none> and password file <none>
I20260806 12:27:57.195680 27251 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:57.195858 27251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:57.196419 27251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:57.198912 27251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "8218ffdcd98c4cc6adbc7366135fdf44"
format_stamp: "Formatted at 2026-08-06 12:27:57 on dist-test-slave-lgr5"
server_key: "093d891bea5ab94336bf1e2a895c2858"
server_key_iv: "5d7c2e6ce34084d851b64364b6e82209"
server_key_version: "encryptionkey@0"
I20260806 12:27:57.199535 27251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "8218ffdcd98c4cc6adbc7366135fdf44"
format_stamp: "Formatted at 2026-08-06 12:27:57 on dist-test-slave-lgr5"
server_key: "093d891bea5ab94336bf1e2a895c2858"
server_key_iv: "5d7c2e6ce34084d851b64364b6e82209"
server_key_version: "encryptionkey@0"
I20260806 12:27:57.203701 27251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 12:27:57.206648 27272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:57.208161 27251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:57.208307 27251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "8218ffdcd98c4cc6adbc7366135fdf44"
format_stamp: "Formatted at 2026-08-06 12:27:57 on dist-test-slave-lgr5"
server_key: "093d891bea5ab94336bf1e2a895c2858"
server_key_iv: "5d7c2e6ce34084d851b64364b6e82209"
server_key_version: "encryptionkey@0"
I20260806 12:27:57.208433 27251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:57.228777 27251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:57.229676 27251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:57.229897 27251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:57.237996 27251 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:36649
I20260806 12:27:57.238085 27350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:36649 every 8 connection(s)
I20260806 12:27:57.239138 27251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:27:57.242868 27351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:57.248306 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 27251
I20260806 12:27:57.248463 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:27:57.248818 29513 external_mini_cluster.cc:1428] Setting key 2317a331c07093691c953400a3760272
I20260806 12:27:57.250394 27351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44: Bootstrap starting.
I20260806 12:27:57.253440 27351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:57.254400 27351 log.cc:824] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:57.257129 27351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44: No bootstrap required, opened a new log
I20260806 12:27:57.260737 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:57.250853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.28.210.126:36649)
0806 12:27:57.251689 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:57.251708 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:57.251751 (+ 43us) server_negotiation.cc:407] Connection header received
0806 12:27:57.252629 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:57.252655 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:57.253199 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:57.253600 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:57.254159 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:57.255370 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:57.256476 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:57.257134 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:57.257313 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:57.258380 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:57.258386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:57.258404 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:57.258941 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:57.259020 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:57.259046 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:57.259154 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:57.259323 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 12:27:57.259673 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":176,"threads_started":1}
I20260806 12:27:57.261767 27351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8218ffdcd98c4cc6adbc7366135fdf44" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36649 } }
I20260806 12:27:57.262197 27351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:57.262255 27351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8218ffdcd98c4cc6adbc7366135fdf44, State: Initialized, Role: FOLLOWER
I20260806 12:27:57.262790 27351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8218ffdcd98c4cc6adbc7366135fdf44" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36649 } }
I20260806 12:27:57.262954 27351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:27:57.263024 27351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:27:57.263149 27351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:57.264667 27351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8218ffdcd98c4cc6adbc7366135fdf44" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36649 } }
I20260806 12:27:57.265172 27351 leader_election.cc:302] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8218ffdcd98c4cc6adbc7366135fdf44; no voters:
I20260806 12:27:57.265570 27351 leader_election.cc:288] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:27:57.265736 27357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:57.266018 27357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [term 1 LEADER]: Becoming Leader. State: Replica: 8218ffdcd98c4cc6adbc7366135fdf44, State: Running, Role: LEADER
I20260806 12:27:57.266474 27357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8218ffdcd98c4cc6adbc7366135fdf44" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36649 } }
I20260806 12:27:57.267302 27351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:27:57.268532 27359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8218ffdcd98c4cc6adbc7366135fdf44. Latest consensus state: current_term: 1 leader_uuid: "8218ffdcd98c4cc6adbc7366135fdf44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8218ffdcd98c4cc6adbc7366135fdf44" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36649 } } }
I20260806 12:27:57.268730 27359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:57.269253 27358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8218ffdcd98c4cc6adbc7366135fdf44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8218ffdcd98c4cc6adbc7366135fdf44" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 36649 } } }
I20260806 12:27:57.269485 27358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44 [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:57.269948 27370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:27:57.274657 27370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:27:57.282356 27370 catalog_manager.cc:1347] Generated new cluster ID: 2e78cf0cc1894d8c9484be99a1412b67
I20260806 12:27:57.282482 27370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:27:57.312361 27370 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:27:57.314049 27370 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:27:57.325763 27370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44: Generated new TSK 0
I20260806 12:27:57.327044 27370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:27:57.336524 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.65:0
--local_ip_for_outbound_sockets=127.28.210.65
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36649
--builtin_ntp_servers=127.28.210.84:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 12:27:57.475209 27386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:57.475536 27386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:57.475600 27386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:57.479945 27386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:57.480036 27386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:57.480145 27386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.65
I20260806 12:27:57.485708 27386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.210.65
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.65
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:57.487130 27386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:57.488958 27386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:57.500638 27397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:57.500690 27394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:57.500634 27393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:57.501034 27386 server_base.cc:1034] running on GCE node
I20260806 12:27:57.501518 27386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:57.502216 27386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:57.503423 27386 hybrid_clock.cc:648] HybridClock initialized: now 1786019277503407 us; error 35 us; skew 500 ppm
I20260806 12:27:57.506093 27386 webserver.cc:460] Webserver started at http://127.28.210.65:34833/ using document root <none> and password file <none>
I20260806 12:27:57.506758 27386 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:57.506842 27386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:57.507107 27386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:57.509586 27386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/data/instance:
uuid: "ac2f2381e8844f849b7847343762b6b3"
format_stamp: "Formatted at 2026-08-06 12:27:57 on dist-test-slave-lgr5"
server_key: "a8eac2a8b92b9ff79288caff0753f88c"
server_key_iv: "fedcb51aa4b34a64dfc7cee74c4fb8f4"
server_key_version: "encryptionkey@0"
I20260806 12:27:57.510195 27386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance:
uuid: "ac2f2381e8844f849b7847343762b6b3"
format_stamp: "Formatted at 2026-08-06 12:27:57 on dist-test-slave-lgr5"
server_key: "a8eac2a8b92b9ff79288caff0753f88c"
server_key_iv: "fedcb51aa4b34a64dfc7cee74c4fb8f4"
server_key_version: "encryptionkey@0"
I20260806 12:27:57.514832 27386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:27:57.519661 27409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:57.521353 27386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:57.521514 27386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/wal
uuid: "ac2f2381e8844f849b7847343762b6b3"
format_stamp: "Formatted at 2026-08-06 12:27:57 on dist-test-slave-lgr5"
server_key: "a8eac2a8b92b9ff79288caff0753f88c"
server_key_iv: "fedcb51aa4b34a64dfc7cee74c4fb8f4"
server_key_version: "encryptionkey@0"
I20260806 12:27:57.521678 27386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:57.553809 27386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:57.554682 27386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:57.554875 27386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:57.555629 27386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:57.556915 27386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:57.556979 27386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:57.557045 27386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:57.557080 27386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:57.566761 27386 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.65:41205
I20260806 12:27:57.566794 27591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.65:41205 every 8 connection(s)
I20260806 12:27:57.567925 27386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/data/info.pb
I20260806 12:27:57.577500 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 27386
I20260806 12:27:57.577652 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-0/wal/instance
I20260806 12:27:57.578111 29513 external_mini_cluster.cc:1428] Setting key 82c0e8829301b5ddb8a2e0d52d79d2a6
I20260806 12:27:57.580351 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:57.570497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:38641 (local address 127.28.210.126:36649)
0806 12:27:57.570650 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:57.570655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:57.572056 (+ 1401us) server_negotiation.cc:407] Connection header received
0806 12:27:57.573859 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:57.573934 (+ 75us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:57.574032 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:57.574126 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:57.575290 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:57.575887 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:57.577518 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:57.577797 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:57.578041 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:57.579708 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:57.579714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:57.579716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:57.579733 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:57.579762 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:57.579774 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:57.579848 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:57.580119 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 12:27:57.580194 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 12:27:57.580801 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.66:0
--local_ip_for_outbound_sockets=127.28.210.66
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36649
--builtin_ntp_servers=127.28.210.84:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 12:27:57.581416 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:57.570877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36649 (local address 127.28.210.65:38641)
0806 12:27:57.571838 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:57.571892 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:57.573551 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:57.574355 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:57.574365 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:57.574470 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:57.575108 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:57.575124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:57.576040 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:57.576044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:57.577337 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:57.577353 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:57.578094 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:57.578098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:57.578317 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:57.579247 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:27:57.579250 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:57.579319 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:27:57.579425 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:57.579864 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:57.579875 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:57.579882 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 12:27:57.580037 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 12:27:57.580388 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":159,"threads_started":1}
I20260806 12:27:57.583204 27592 heartbeater.cc:347] Connected to a master server at 127.28.210.126:36649
I20260806 12:27:57.583494 27592 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:57.584182 27592 heartbeater.cc:511] Master 127.28.210.126:36649 requested a full tablet report, sending...
I20260806 12:27:57.586169 27295 ts_manager.cc:194] Registered new tserver with Master: ac2f2381e8844f849b7847343762b6b3 (127.28.210.65:41205)
I20260806 12:27:57.587910 27295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.210.65:38641
W20260806 12:27:57.725996 27596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:57.726300 27596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:57.726364 27596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:57.730563 27596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:57.730638 27596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:57.730722 27596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.66
I20260806 12:27:57.735863 27596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.210.66
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.66
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:57.737123 27596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:57.738698 27596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:57.750150 27604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:57.750267 27606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:57.750231 27596 server_base.cc:1034] running on GCE node
W20260806 12:27:57.750159 27603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:57.750921 27596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:57.751596 27596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:57.752759 27596 hybrid_clock.cc:648] HybridClock initialized: now 1786019277752741 us; error 36 us; skew 500 ppm
I20260806 12:27:57.754797 27596 webserver.cc:460] Webserver started at http://127.28.210.66:36731/ using document root <none> and password file <none>
I20260806 12:27:57.755594 27596 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:57.755692 27596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:57.755949 27596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:57.758445 27596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/data/instance:
uuid: "5416db4f31754d1482a055ca8296098d"
format_stamp: "Formatted at 2026-08-06 12:27:57 on dist-test-slave-lgr5"
server_key: "4a24d285d8d4643402f691a358336a67"
server_key_iv: "7a7e4c3127210dc6620d23155be1eee3"
server_key_version: "encryptionkey@0"
I20260806 12:27:57.759155 27596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance:
uuid: "5416db4f31754d1482a055ca8296098d"
format_stamp: "Formatted at 2026-08-06 12:27:57 on dist-test-slave-lgr5"
server_key: "4a24d285d8d4643402f691a358336a67"
server_key_iv: "7a7e4c3127210dc6620d23155be1eee3"
server_key_version: "encryptionkey@0"
I20260806 12:27:57.763664 27596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 12:27:57.766422 27614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:57.768155 27596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 12:27:57.768324 27596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/wal
uuid: "5416db4f31754d1482a055ca8296098d"
format_stamp: "Formatted at 2026-08-06 12:27:57 on dist-test-slave-lgr5"
server_key: "4a24d285d8d4643402f691a358336a67"
server_key_iv: "7a7e4c3127210dc6620d23155be1eee3"
server_key_version: "encryptionkey@0"
I20260806 12:27:57.768463 27596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:57.787271 27596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:57.788143 27596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:57.788337 27596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:57.789031 27596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:57.790194 27596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:57.790251 27596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:57.790310 27596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:57.790336 27596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:57.799341 27596 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.66:37573
I20260806 12:27:57.799369 27792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.66:37573 every 8 connection(s)
I20260806 12:27:57.800380 27596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/data/info.pb
I20260806 12:27:57.801220 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 27596
I20260806 12:27:57.801334 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/wal/instance
I20260806 12:27:57.801621 29513 external_mini_cluster.cc:1428] Setting key 600ef8aff2fe4e1e28dcbb897219404d
I20260806 12:27:57.804001 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.210.67:0
--local_ip_for_outbound_sockets=127.28.210.67
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36649
--builtin_ntp_servers=127.28.210.84:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 12:27:57.810945 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:57.802375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.66:48937 (local address 127.28.210.126:36649)
0806 12:27:57.802516 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:57.802520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:57.804047 (+ 1527us) server_negotiation.cc:407] Connection header received
0806 12:27:57.805872 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:57.805875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:57.805934 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:57.806033 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:57.807060 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:57.807717 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:57.808972 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:57.809197 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:57.809291 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:57.810456 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:57.810460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:57.810462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:57.810476 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:57.810497 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:57.810505 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:57.810564 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:57.810713 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 12:27:57.810782 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 12:27:57.811895 27799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:57.802965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36649 (local address 127.28.210.66:48937)
0806 12:27:57.803839 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:57.803919 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:57.805600 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:57.806252 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:57.806260 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:57.806348 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:57.806906 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:57.806920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:57.807826 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:57.807828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:57.808830 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:57.808842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:57.809367 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:57.809372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:57.809557 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:57.810175 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:27:57.810177 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:57.810221 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:27:57.810296 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:57.810576 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:57.810587 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:57.810593 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 12:27:57.810705 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 12:27:57.811040 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":156,"threads_started":1}
I20260806 12:27:57.813149 27795 heartbeater.cc:347] Connected to a master server at 127.28.210.126:36649
I20260806 12:27:57.813478 27795 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:57.814204 27795 heartbeater.cc:511] Master 127.28.210.126:36649 requested a full tablet report, sending...
I20260806 12:27:57.815512 27295 ts_manager.cc:194] Registered new tserver with Master: 5416db4f31754d1482a055ca8296098d (127.28.210.66:37573)
I20260806 12:27:57.816331 27295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.210.66:48937
W20260806 12:27:57.925541 27800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:57.925879 27800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:57.925925 27800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:57.930178 27800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:57.930253 27800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:57.930312 27800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.210.67
I20260806 12:27:57.935398 27800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.210.67
--webserver_port=0
--tserver_master_addrs=127.28.210.126:36649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.210.67
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:57.936681 27800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:57.938210 27800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:27:57.949218 27812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:57.949218 27813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:57.949249 27815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:57.949239 27800 server_base.cc:1034] running on GCE node
I20260806 12:27:57.949994 27800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:57.950718 27800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:57.951921 27800 hybrid_clock.cc:648] HybridClock initialized: now 1786019277951899 us; error 44 us; skew 500 ppm
I20260806 12:27:57.954133 27800 webserver.cc:460] Webserver started at http://127.28.210.67:44213/ using document root <none> and password file <none>
I20260806 12:27:57.954751 27800 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:57.954833 27800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:57.955094 27800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:57.957391 27800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/data/instance:
uuid: "9a91b258bcd047a5bfa56a682bb2fb36"
format_stamp: "Formatted at 2026-08-06 12:27:57 on dist-test-slave-lgr5"
server_key: "919a5e41fde5511c342e3e2e5d1e88ea"
server_key_iv: "575a7cc3081d702b33925066dd2fb485"
server_key_version: "encryptionkey@0"
I20260806 12:27:57.957988 27800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance:
uuid: "9a91b258bcd047a5bfa56a682bb2fb36"
format_stamp: "Formatted at 2026-08-06 12:27:57 on dist-test-slave-lgr5"
server_key: "919a5e41fde5511c342e3e2e5d1e88ea"
server_key_iv: "575a7cc3081d702b33925066dd2fb485"
server_key_version: "encryptionkey@0"
I20260806 12:27:57.962028 27800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:27:57.964802 27825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:57.966142 27800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:27:57.966269 27800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/wal
uuid: "9a91b258bcd047a5bfa56a682bb2fb36"
format_stamp: "Formatted at 2026-08-06 12:27:57 on dist-test-slave-lgr5"
server_key: "919a5e41fde5511c342e3e2e5d1e88ea"
server_key_iv: "575a7cc3081d702b33925066dd2fb485"
server_key_version: "encryptionkey@0"
I20260806 12:27:57.966388 27800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:57.982053 27800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:57.983003 27800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:57.983219 27800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:57.983902 27800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 12:27:57.985049 27800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 12:27:57.985102 27800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:57.985158 27800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 12:27:57.985188 27800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:57.994697 27800 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.67:45341
I20260806 12:27:57.994720 27998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.67:45341 every 8 connection(s)
I20260806 12:27:57.995731 27800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/data/info.pb
I20260806 12:27:58.003123 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 27800
I20260806 12:27:58.003290 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/wal/instance
I20260806 12:27:58.003691 29513 external_mini_cluster.cc:1428] Setting key bbb0746bd7cf7b361e0414047734a2c0
I20260806 12:27:58.008148 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:57.997831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.67:41301 (local address 127.28.210.126:36649)
0806 12:27:57.998006 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:57.998010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:57.999216 (+ 1206us) server_negotiation.cc:407] Connection header received
0806 12:27:58.001104 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:58.001108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:58.001199 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:58.001351 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:58.002506 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.003202 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.004968 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.005388 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.005502 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.007499 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:58.007504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:58.007507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:58.007526 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:58.007554 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:58.007565 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:58.007648 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:58.007864 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 12:27:58.007947 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 12:27:58.009321 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:57.998244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:36649 (local address 127.28.210.67:41301)
0806 12:27:57.999041 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:57.999088 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:58.000692 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:58.001603 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:58.001615 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:58.001739 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:27:58.002334 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:58.002351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.003410 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.003415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:58.004764 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:58.004781 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.005781 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.005794 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:58.006070 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.007070 (+ 1000us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 12:27:58.007072 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:27:58.007150 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 12:27:58.007276 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:27:58.007663 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:27:58.007676 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:27:58.007684 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 12:27:58.007848 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 12:27:58.008288 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":137,"threads_started":1}
I20260806 12:27:58.010700 27999 heartbeater.cc:347] Connected to a master server at 127.28.210.126:36649
I20260806 12:27:58.011030 27999 heartbeater.cc:464] Registering TS with master...
I20260806 12:27:58.011742 27999 heartbeater.cc:511] Master 127.28.210.126:36649 requested a full tablet report, sending...
I20260806 12:27:58.013199 27295 ts_manager.cc:194] Registered new tserver with Master: 9a91b258bcd047a5bfa56a682bb2fb36 (127.28.210.67:45341)
I20260806 12:27:58.014053 27295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.210.67:41301
I20260806 12:27:58.019452 29513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 12:27:58.025439 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.021586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.28.210.126:36649)
0806 12:27:58.021703 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:58.021707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:58.021895 (+ 188us) server_negotiation.cc:407] Connection header received
0806 12:27:58.022027 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:58.022029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:58.022072 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:58.022142 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:58.022461 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.022973 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.023896 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.024174 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.024261 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.025064 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:58.025067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:58.025069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:27:58.025083 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:58.025101 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:27:58.025109 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:58.025167 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:58.025229 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 12:27:58.025290 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 12:27:58.029161 27295 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 12:27:58.032395 27295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 12:27:58.049794 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.042815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:41205 (local address 127.0.0.1:37282)
0806 12:27:58.044147 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:58.044219 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:58.044376 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:58.045029 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:58.045037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:58.045061 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:58.045262 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:58.045270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.046553 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.046556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:58.047864 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:58.047876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.049069 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.049073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:58.049220 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.049279 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 12:27:58.049414 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 12:27:58.049602 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"thread_start_us":146,"threads_started":1}
I20260806 12:27:58.050021 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.042939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.28.210.65:41205)
0806 12:27:58.043849 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:58.043856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:58.044538 (+ 682us) server_negotiation.cc:407] Connection header received
0806 12:27:58.044634 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:58.044638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:58.044830 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:58.044986 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:58.045413 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.046405 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.048043 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.048902 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.049046 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.049321 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:58.049613 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 12:27:58.049797 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":120,"threads_started":1}
I20260806 12:27:58.050434 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.042921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:45341 (local address 127.0.0.1:60450)
0806 12:27:58.044147 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:58.044219 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:58.044379 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:58.045244 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:58.045247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:58.045268 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:58.045380 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:58.045388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.046996 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.047000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:58.048332 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:58.048346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.049931 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.049935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:58.050037 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.050065 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 12:27:58.050163 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 12:27:58.050270 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":17,"thread_start_us":148,"threads_started":1}
I20260806 12:27:58.050570 28027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.043978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48464 (local address 127.28.210.66:37573)
0806 12:27:58.044844 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:58.044856 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:58.045116 (+ 260us) server_negotiation.cc:407] Connection header received
0806 12:27:58.045229 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:58.045234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:58.045510 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:58.045690 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:58.045929 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.047110 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.048585 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.049281 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.049407 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.049614 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:58.050259 (+ 645us) server_negotiation.cc:299] Negotiation successful
0806 12:27:58.050389 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":179,"threads_started":1}
I20260806 12:27:58.050107 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.044408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:37573 (local address 127.0.0.1:48464)
0806 12:27:58.044875 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:58.044888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:58.045004 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:58.045679 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:58.045682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:58.045704 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:58.045790 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:58.045796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.047243 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.047246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:58.048439 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:58.048458 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.049687 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.049691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:58.049779 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.049802 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 12:27:58.049876 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 12:27:58.049973 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":51,"thread_start_us":102,"threads_started":1}
I20260806 12:27:58.051167 28026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.043190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.28.210.67:45341)
0806 12:27:58.043759 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:58.043774 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:58.044318 (+ 544us) server_negotiation.cc:407] Connection header received
0806 12:27:58.044581 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:58.044588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:58.044830 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:58.044997 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:58.045554 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.046814 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.048513 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.049341 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.049463 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.049835 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:58.050858 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0806 12:27:58.050992 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":165,"threads_started":1}
I20260806 12:27:58.054409 27483 tablet_service.cc:1467] Processing CreateTablet for tablet 8cef86d69c7f4631a54c90058bf1290d (DEFAULT_TABLE table=test-table [id=9c6a67fba8ca45e8b120f52dcd47003f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:58.054409 27693 tablet_service.cc:1467] Processing CreateTablet for tablet 8cef86d69c7f4631a54c90058bf1290d (DEFAULT_TABLE table=test-table [id=9c6a67fba8ca45e8b120f52dcd47003f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:58.054881 27895 tablet_service.cc:1467] Processing CreateTablet for tablet 8cef86d69c7f4631a54c90058bf1290d (DEFAULT_TABLE table=test-table [id=9c6a67fba8ca45e8b120f52dcd47003f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 12:27:58.056027 27895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cef86d69c7f4631a54c90058bf1290d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:58.056051 27693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cef86d69c7f4631a54c90058bf1290d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:58.056052 27483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cef86d69c7f4631a54c90058bf1290d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:58.064493 28033 tablet_bootstrap.cc:492] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36: Bootstrap starting.
I20260806 12:27:58.065451 28034 tablet_bootstrap.cc:492] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3: Bootstrap starting.
I20260806 12:27:58.067677 28034 tablet_bootstrap.cc:654] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:58.067677 28033 tablet_bootstrap.cc:654] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:58.068639 28035 tablet_bootstrap.cc:492] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d: Bootstrap starting.
I20260806 12:27:58.068820 28033 log.cc:824] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:58.069466 28034 log.cc:824] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:58.071375 28035 tablet_bootstrap.cc:654] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:58.071813 28034 tablet_bootstrap.cc:492] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3: No bootstrap required, opened a new log
I20260806 12:27:58.071813 28033 tablet_bootstrap.cc:492] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36: No bootstrap required, opened a new log
I20260806 12:27:58.072115 28033 ts_tablet_manager.cc:1397] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 12:27:58.072115 28034 ts_tablet_manager.cc:1397] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 12:27:58.072436 28035 log.cc:824] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:58.074671 28035 tablet_bootstrap.cc:492] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d: No bootstrap required, opened a new log
I20260806 12:27:58.074929 28035 ts_tablet_manager.cc:1397] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 12:27:58.077199 28034 raft_consensus.cc:348] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2f2381e8844f849b7847343762b6b3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 41205 } } peers { permanent_uuid: "9a91b258bcd047a5bfa56a682bb2fb36" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45341 } } peers { permanent_uuid: "5416db4f31754d1482a055ca8296098d" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37573 } }
I20260806 12:27:58.077199 28033 raft_consensus.cc:348] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2f2381e8844f849b7847343762b6b3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 41205 } } peers { permanent_uuid: "9a91b258bcd047a5bfa56a682bb2fb36" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45341 } } peers { permanent_uuid: "5416db4f31754d1482a055ca8296098d" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37573 } }
I20260806 12:27:58.077543 28033 raft_consensus.cc:374] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:58.077543 28034 raft_consensus.cc:374] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:58.077607 28033 raft_consensus.cc:729] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a91b258bcd047a5bfa56a682bb2fb36, State: Initialized, Role: FOLLOWER
I20260806 12:27:58.077607 28034 raft_consensus.cc:729] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac2f2381e8844f849b7847343762b6b3, State: Initialized, Role: FOLLOWER
I20260806 12:27:58.078140 28033 consensus_queue.cc:260] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2f2381e8844f849b7847343762b6b3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 41205 } } peers { permanent_uuid: "9a91b258bcd047a5bfa56a682bb2fb36" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45341 } } peers { permanent_uuid: "5416db4f31754d1482a055ca8296098d" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37573 } }
I20260806 12:27:58.078140 28034 consensus_queue.cc:260] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2f2381e8844f849b7847343762b6b3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 41205 } } peers { permanent_uuid: "9a91b258bcd047a5bfa56a682bb2fb36" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45341 } } peers { permanent_uuid: "5416db4f31754d1482a055ca8296098d" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37573 } }
I20260806 12:27:58.079245 27999 heartbeater.cc:503] Master 127.28.210.126:36649 was elected leader, sending a full tablet report...
I20260806 12:27:58.079386 28034 ts_tablet_manager.cc:1428] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260806 12:27:58.079386 28033 ts_tablet_manager.cc:1428] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260806 12:27:58.079340 28035 raft_consensus.cc:348] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2f2381e8844f849b7847343762b6b3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 41205 } } peers { permanent_uuid: "9a91b258bcd047a5bfa56a682bb2fb36" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45341 } } peers { permanent_uuid: "5416db4f31754d1482a055ca8296098d" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37573 } }
I20260806 12:27:58.079618 28035 raft_consensus.cc:374] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:58.079691 28035 raft_consensus.cc:729] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5416db4f31754d1482a055ca8296098d, State: Initialized, Role: FOLLOWER
I20260806 12:27:58.079766 27592 heartbeater.cc:503] Master 127.28.210.126:36649 was elected leader, sending a full tablet report...
I20260806 12:27:58.080221 28035 consensus_queue.cc:260] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2f2381e8844f849b7847343762b6b3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 41205 } } peers { permanent_uuid: "9a91b258bcd047a5bfa56a682bb2fb36" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45341 } } peers { permanent_uuid: "5416db4f31754d1482a055ca8296098d" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37573 } }
I20260806 12:27:58.081198 27795 heartbeater.cc:503] Master 127.28.210.126:36649 was elected leader, sending a full tablet report...
I20260806 12:27:58.081312 28035 ts_tablet_manager.cc:1428] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 12:27:58.111500 28040 raft_consensus.cc:482] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 12:27:58.111835 28040 raft_consensus.cc:504] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2f2381e8844f849b7847343762b6b3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 41205 } } peers { permanent_uuid: "9a91b258bcd047a5bfa56a682bb2fb36" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45341 } } peers { permanent_uuid: "5416db4f31754d1482a055ca8296098d" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37573 } }
I20260806 12:27:58.113276 28040 leader_election.cc:288] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a91b258bcd047a5bfa56a682bb2fb36 (127.28.210.67:45341), 5416db4f31754d1482a055ca8296098d (127.28.210.66:37573)
I20260806 12:27:58.118462 28027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.113832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:37729 (local address 127.28.210.66:37573)
0806 12:27:58.113981 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:58.113986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:58.114222 (+ 236us) server_negotiation.cc:407] Connection header received
0806 12:27:58.114367 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:58.114370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:58.114420 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:58.114496 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:58.115405 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.115965 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.117381 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.117813 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.117938 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.118020 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:58.118218 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 12:27:58.118307 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:27:58.118468 28026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.113777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.210.65:43811 (local address 127.28.210.67:45341)
0806 12:27:58.113938 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:58.113944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:58.114169 (+ 225us) server_negotiation.cc:407] Connection header received
0806 12:27:58.114336 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:58.114340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:58.114400 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:58.114466 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 12:27:58.115080 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.115844 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.117393 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.117844 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.117974 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.118032 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:58.118239 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 12:27:58.118321 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 12:27:58.118491 28045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.113591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.66:37573 (local address 127.28.210.65:37729)
0806 12:27:58.114094 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:58.114107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:58.114250 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:58.114524 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:58.114527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:58.114781 (+ 254us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:58.115043 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:58.115059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.116139 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.116142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:58.117217 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:58.117233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.117967 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.117971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:58.118085 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.118112 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 12:27:58.118181 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 12:27:58.118292 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":71,"spinlock_wait_cycles":9088,"thread_start_us":150,"threads_started":1}
I20260806 12:27:58.118507 28044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.113591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.67:45341 (local address 127.28.210.65:43811)
0806 12:27:58.114055 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:58.114069 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 12:27:58.114204 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:27:58.114517 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:27:58.114520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:27:58.114538 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 12:27:58.114873 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:58.114881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.116018 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.116022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:58.117223 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:27:58.117238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.118000 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.118004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:27:58.118110 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.118130 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 12:27:58.118211 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 12:27:58.118345 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":71,"spinlock_wait_cycles":377216,"thread_start_us":120,"threads_started":1}
I20260806 12:27:58.119244 27717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cef86d69c7f4631a54c90058bf1290d" candidate_uuid: "ac2f2381e8844f849b7847343762b6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5416db4f31754d1482a055ca8296098d" is_pre_election: true
I20260806 12:27:58.119530 27927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cef86d69c7f4631a54c90058bf1290d" candidate_uuid: "ac2f2381e8844f849b7847343762b6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a91b258bcd047a5bfa56a682bb2fb36" is_pre_election: true
I20260806 12:27:58.119683 27717 raft_consensus.cc:2454] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac2f2381e8844f849b7847343762b6b3 in term 0.
I20260806 12:27:58.119818 27927 raft_consensus.cc:2454] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac2f2381e8844f849b7847343762b6b3 in term 0.
I20260806 12:27:58.120421 27413 leader_election.cc:302] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5416db4f31754d1482a055ca8296098d, ac2f2381e8844f849b7847343762b6b3; no voters:
I20260806 12:27:58.120738 28040 raft_consensus.cc:2781] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 12:27:58.120850 28040 raft_consensus.cc:482] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 12:27:58.120906 28040 raft_consensus.cc:3037] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:58.122157 28040 raft_consensus.cc:504] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2f2381e8844f849b7847343762b6b3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 41205 } } peers { permanent_uuid: "9a91b258bcd047a5bfa56a682bb2fb36" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45341 } } peers { permanent_uuid: "5416db4f31754d1482a055ca8296098d" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37573 } }
I20260806 12:27:58.122579 28040 leader_election.cc:288] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [CANDIDATE]: Term 1 election: Requested vote from peers 9a91b258bcd047a5bfa56a682bb2fb36 (127.28.210.67:45341), 5416db4f31754d1482a055ca8296098d (127.28.210.66:37573)
I20260806 12:27:58.123016 27927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cef86d69c7f4631a54c90058bf1290d" candidate_uuid: "ac2f2381e8844f849b7847343762b6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a91b258bcd047a5bfa56a682bb2fb36"
I20260806 12:27:58.123085 27717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cef86d69c7f4631a54c90058bf1290d" candidate_uuid: "ac2f2381e8844f849b7847343762b6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5416db4f31754d1482a055ca8296098d"
I20260806 12:27:58.123142 27927 raft_consensus.cc:3037] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:58.123266 27717 raft_consensus.cc:3037] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:58.124419 27927 raft_consensus.cc:2454] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac2f2381e8844f849b7847343762b6b3 in term 1.
I20260806 12:27:58.124516 27717 raft_consensus.cc:2454] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac2f2381e8844f849b7847343762b6b3 in term 1.
I20260806 12:27:58.124894 27413 leader_election.cc:302] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5416db4f31754d1482a055ca8296098d, ac2f2381e8844f849b7847343762b6b3; no voters:
I20260806 12:27:58.125092 28040 raft_consensus.cc:2781] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:58.125370 28040 raft_consensus.cc:686] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [term 1 LEADER]: Becoming Leader. State: Replica: ac2f2381e8844f849b7847343762b6b3, State: Running, Role: LEADER
I20260806 12:27:58.125775 28040 consensus_queue.cc:237] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2f2381e8844f849b7847343762b6b3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 41205 } } peers { permanent_uuid: "9a91b258bcd047a5bfa56a682bb2fb36" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45341 } } peers { permanent_uuid: "5416db4f31754d1482a055ca8296098d" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37573 } }
I20260806 12:27:58.129383 27288 catalog_manager.cc:5495] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac2f2381e8844f849b7847343762b6b3 (127.28.210.65). New cstate: current_term: 1 leader_uuid: "ac2f2381e8844f849b7847343762b6b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac2f2381e8844f849b7847343762b6b3" member_type: VOTER last_known_addr { host: "127.28.210.65" port: 41205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a91b258bcd047a5bfa56a682bb2fb36" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5416db4f31754d1482a055ca8296098d" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37573 } health_report { overall_health: UNKNOWN } } }
I20260806 12:27:58.156270 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.152150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.28.210.65:41205)
0806 12:27:58.152329 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:58.152333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:58.152350 (+ 17us) server_negotiation.cc:407] Connection header received
0806 12:27:58.152447 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:58.152449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:58.152518 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:58.152619 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:27:58.152915 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.153489 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.154454 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:58.154686 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:58.154780 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:58.155152 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:27:58.155696 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:27:58.155798 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:58.155922 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 12:27:58.155982 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 12:27:58.163399 27927 raft_consensus.cc:1261] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36 [term 1 FOLLOWER]: Refusing update from remote peer ac2f2381e8844f849b7847343762b6b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:58.163399 27717 raft_consensus.cc:1261] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d [term 1 FOLLOWER]: Refusing update from remote peer ac2f2381e8844f849b7847343762b6b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:27:58.164161 28046 consensus_queue.cc:1035] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a91b258bcd047a5bfa56a682bb2fb36" member_type: VOTER last_known_addr { host: "127.28.210.67" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:58.164359 28040 consensus_queue.cc:1035] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5416db4f31754d1482a055ca8296098d" member_type: VOTER last_known_addr { host: "127.28.210.66" port: 37573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:27:58.174232 28052 mvcc.cc:205] Tried to move back new op lower bound from 7315534963348320256 to 7315534963206406144. Current Snapshot: MvccSnapshot[applied={T|T < 7315534963348320256}]
I20260806 12:27:58.174288 28051 mvcc.cc:205] Tried to move back new op lower bound from 7315534963348320256 to 7315534963206406144. Current Snapshot: MvccSnapshot[applied={T|T < 7315534963348320256}]
I20260806 12:27:58.174947 28050 mvcc.cc:205] Tried to move back new op lower bound from 7315534963348320256 to 7315534963206406144. Current Snapshot: MvccSnapshot[applied={T|T < 7315534963348320256}]
I20260806 12:27:58.184929 27294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:58.185134 27294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 12:27:58.188184 27294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8218ffdcd98c4cc6adbc7366135fdf44: Sending DeleteTablet for 3 replicas of tablet 8cef86d69c7f4631a54c90058bf1290d
I20260806 12:27:58.189182 27895 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cef86d69c7f4631a54c90058bf1290d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:58 UTC) from {username='slave'} at 127.0.0.1:60450
I20260806 12:27:58.189284 27693 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cef86d69c7f4631a54c90058bf1290d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:58 UTC) from {username='slave'} at 127.0.0.1:48464
I20260806 12:27:58.189465 27483 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cef86d69c7f4631a54c90058bf1290d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 12:27:58 UTC) from {username='slave'} at 127.0.0.1:37282
I20260806 12:27:58.190027 28071 tablet_replica.cc:330] T 8cef86d69c7f4631a54c90058bf1290d P ac2f2381e8844f849b7847343762b6b3: stopping tablet replica
I20260806 12:27:58.190341 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 27386
I20260806 12:27:58.221217 28069 tablet_replica.cc:330] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36: stopping tablet replica
I20260806 12:27:58.221580 28069 raft_consensus.cc:2229] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:27:58.221907 28069 raft_consensus.cc:2258] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:58.222311 28070 tablet_replica.cc:330] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d: stopping tablet replica
I20260806 12:27:58.222607 28070 raft_consensus.cc:2229] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 12:27:58.222892 28070 raft_consensus.cc:2258] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 12:27:58.223999 28069 ts_tablet_manager.cc:1905] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:58.224278 28070 ts_tablet_manager.cc:1905] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 12:27:58.226652 28070 ts_tablet_manager.cc:1918] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:58.226768 28070 log.cc:1196] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-1/wal/wals/8cef86d69c7f4631a54c90058bf1290d
I20260806 12:27:58.227063 28070 ts_tablet_manager.cc:1939] T 8cef86d69c7f4631a54c90058bf1290d P 5416db4f31754d1482a055ca8296098d: Deleting consensus metadata
I20260806 12:27:58.227057 28069 ts_tablet_manager.cc:1918] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 12:27:58.227161 28069 log.cc:1196] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36: Deleting WAL directory at /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786018995816160-29513-0/minicluster-data/ts-2/wal/wals/8cef86d69c7f4631a54c90058bf1290d
I20260806 12:27:58.227470 28069 ts_tablet_manager.cc:1939] T 8cef86d69c7f4631a54c90058bf1290d P 9a91b258bcd047a5bfa56a682bb2fb36: Deleting consensus metadata
I20260806 12:27:58.228837 27273 catalog_manager.cc:4841] TS 9a91b258bcd047a5bfa56a682bb2fb36 (127.28.210.67:45341): tablet 8cef86d69c7f4631a54c90058bf1290d (table test-table [id=9c6a67fba8ca45e8b120f52dcd47003f]) successfully deleted
I20260806 12:27:58.228783 27276 catalog_manager.cc:4841] TS 5416db4f31754d1482a055ca8296098d (127.28.210.66:37573): tablet 8cef86d69c7f4631a54c90058bf1290d (table test-table [id=9c6a67fba8ca45e8b120f52dcd47003f]) successfully deleted
W20260806 12:27:58.229839 27276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.210.65:41205 (error 108)
I20260806 12:27:58.230132 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 27596
I20260806 12:27:58.242669 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:58.242196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.65:41205 (local address 127.0.0.1:37304)
0806 12:27:58.242473 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 12:27:58.242555 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.65:41205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
W20260806 12:27:58.244251 27276 catalog_manager.cc:4584] TS ac2f2381e8844f849b7847343762b6b3 (127.28.210.65:41205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8cef86d69c7f4631a54c90058bf1290d: Network error: Client connection negotiation failed: client connection to 127.28.210.65:41205: connect: Connection refused (error 111)
I20260806 12:27:58.262821 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 27800
I20260806 12:27:58.293713 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 27251
2026-08-06T12:27:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1300 ms)
[----------] 6 tests from EncryptionPolicyTest (12794 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:27:58 dist-test-slave-lgr5 krb5kdc[28080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:27:58 dist-test-slave-lgr5 krb5kdc[28080](info): set up 2 sockets
Aug 06 12:27:58 dist-test-slave-lgr5 krb5kdc[28080](info): commencing operation
krb5kdc: starting...
W20260806 12:27:59.484447 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:27:59 dist-test-slave-lgr5 krb5kdc[28080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:27:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:27:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 12:27:59.633285 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:27:59.761004 28098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:27:59.761355 28098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:27:59.761420 28098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:27:59.765830 28098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:27:59.765924 28098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:27:59.765955 28098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:27:59.765978 28098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:27:59.766000 28098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:27:59.772034 28098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:27:59.773886 28098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:27:59.775218 28098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:27:59.785537 28098 server_base.cc:1034] running on GCE node
W20260806 12:27:59.785567 28104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:59.785595 28108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:27:59.785544 28105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:27:59.786464 28098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:27:59.787681 28098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:27:59.788877 28098 hybrid_clock.cc:648] HybridClock initialized: now 1786019279788846 us; error 44 us; skew 500 ppm
Aug 06 12:27:59 dist-test-slave-lgr5 krb5kdc[28080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:27:59.791953 28098 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 12:27:59.793079 28098 webserver.cc:460] Webserver started at http://127.0.0.1:35751/ using document root <none> and password file <none>
I20260806 12:27:59.793700 28098 fs_manager.cc:359] Metadata directory not provided
I20260806 12:27:59.793766 28098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:27:59.793959 28098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:27:59.796563 28098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "3d2ab993ab7240a4993282d04d76c03f"
format_stamp: "Formatted at 2026-08-06 12:27:59 on dist-test-slave-lgr5"
server_key: "8d01bb5b1915426676deb29bbf24822b"
server_key_iv: "18e6182a307e64d8021be4ee6b15bc53"
server_key_version: "encryptionkey@0"
I20260806 12:27:59.797358 28098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "3d2ab993ab7240a4993282d04d76c03f"
format_stamp: "Formatted at 2026-08-06 12:27:59 on dist-test-slave-lgr5"
server_key: "8d01bb5b1915426676deb29bbf24822b"
server_key_iv: "18e6182a307e64d8021be4ee6b15bc53"
server_key_version: "encryptionkey@0"
I20260806 12:27:59.801802 28098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:27:59.804917 28120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:27:59.806564 28098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:27:59.806723 28098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "3d2ab993ab7240a4993282d04d76c03f"
format_stamp: "Formatted at 2026-08-06 12:27:59 on dist-test-slave-lgr5"
server_key: "8d01bb5b1915426676deb29bbf24822b"
server_key_iv: "18e6182a307e64d8021be4ee6b15bc53"
server_key_version: "encryptionkey@0"
I20260806 12:27:59.806856 28098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:27:59.836946 28098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:27:59.841159 28098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:27:59.841356 28098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:27:59.849419 28098 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36905
I20260806 12:27:59.849444 28213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36905 every 8 connection(s)
I20260806 12:27:59.850672 28098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:27:59.852722 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 28098
I20260806 12:27:59.852885 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:27:59.853188 29513 external_mini_cluster.cc:1428] Setting key a72b9171333f684c5cf498b1950ea801
I20260806 12:27:59.854327 28214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:27:59.860733 28214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f: Bootstrap starting.
Aug 06 12:27:59 dist-test-slave-lgr5 krb5kdc[28080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 12:27:59.863754 28214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f: Neither blocks nor log segments found. Creating new log.
I20260806 12:27:59.864848 28214 log.cc:824] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f: Log is configured to *not* fsync() on all Append() calls
I20260806 12:27:59.867636 28214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f: No bootstrap required, opened a new log
I20260806 12:27:59.869330 28218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:59.854737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44663 (local address 127.0.0.1:36905)
0806 12:27:59.855243 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:59.855265 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:59.855449 (+ 184us) server_negotiation.cc:407] Connection header received
0806 12:27:59.856296 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:59.856318 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:59.856697 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:59.857032 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:59.857853 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:59.858877 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:59.859749 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:59.860290 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:59.860441 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:59.863388 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:59.863395 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:59.863415 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:59.863457 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:59.865561 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:59.866214 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:59.866220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:59.866227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:59.866383 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:59.866692 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:59.866695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:59.866697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:59.867278 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:59.867514 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:59.867893 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 12:27:59.868213 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":146,"threads_started":1}
I20260806 12:27:59.872577 28214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2ab993ab7240a4993282d04d76c03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36905 } }
I20260806 12:27:59.872897 28214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:27:59.872957 28214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2ab993ab7240a4993282d04d76c03f, State: Initialized, Role: FOLLOWER
I20260806 12:27:59.873504 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:27:59.873486 28214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2ab993ab7240a4993282d04d76c03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36905 } }
I20260806 12:27:59.873667 28214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:27:59.873737 28214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:27:59.873858 28214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:27:59.875442 28214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2ab993ab7240a4993282d04d76c03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36905 } }
I20260806 12:27:59.875921 28214 leader_election.cc:302] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d2ab993ab7240a4993282d04d76c03f; no voters:
I20260806 12:27:59.876415 28214 leader_election.cc:288] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:27:59.876503 28226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:27:59.876720 28226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [term 1 LEADER]: Becoming Leader. State: Replica: 3d2ab993ab7240a4993282d04d76c03f, State: Running, Role: LEADER
I20260806 12:27:59.877112 28226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2ab993ab7240a4993282d04d76c03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36905 } }
I20260806 12:27:59.878144 28214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:27:59.879307 28227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d2ab993ab7240a4993282d04d76c03f. Latest consensus state: current_term: 1 leader_uuid: "3d2ab993ab7240a4993282d04d76c03f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2ab993ab7240a4993282d04d76c03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36905 } } }
I20260806 12:27:59.879500 28227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:59.879875 28229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d2ab993ab7240a4993282d04d76c03f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2ab993ab7240a4993282d04d76c03f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36905 } } }
I20260806 12:27:59.880002 28229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f [sys.catalog]: This master's current role is: LEADER
I20260806 12:27:59.880694 28241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:27:59.884428 28241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:27:59.887830 28218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:27:59.876258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53305 (local address 127.0.0.1:36905)
0806 12:27:59.876416 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 12:27:59.876422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:27:59.876629 (+ 207us) server_negotiation.cc:407] Connection header received
0806 12:27:59.876846 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:27:59.876850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:27:59.876911 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:27:59.876969 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:27:59.878066 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:59.879061 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:59.880357 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:27:59.880748 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:27:59.880866 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:27:59.883533 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:27:59.883537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:27:59.883541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:27:59.883576 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:27:59.885401 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:59.886140 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:59.886144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:59.886146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:59.886253 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:27:59.886726 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:27:59.886730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:27:59.886732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:27:59.887026 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:27:59.887198 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 12:27:59.887534 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 12:27:59.887661 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 12:27:59.891903 28241 catalog_manager.cc:1347] Generated new cluster ID: b251319955784188a13a28c31db6bc8e
I20260806 12:27:59.892009 28241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:27:59.904172 28241 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:27:59.905961 28241 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:27:59.919417 28241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d2ab993ab7240a4993282d04d76c03f: Generated new TSK 0
I20260806 12:27:59.920327 28241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:27:59.928668 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 28098
W20260806 12:27:59.946516 28078 connection.cc:535] client connection to 127.0.0.1:36905 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36905): Connection reset by peer (error 104)
2026-08-06T12:27:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1632 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:00 dist-test-slave-lgr5 krb5kdc[28256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:00 dist-test-slave-lgr5 krb5kdc[28256](info): set up 2 sockets
Aug 06 12:28:00 dist-test-slave-lgr5 krb5kdc[28256](info): commencing operation
krb5kdc: starting...
W20260806 12:28:01.126139 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:01 dist-test-slave-lgr5 krb5kdc[28256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 12:28:01.284588 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:01.418556 28278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:01.418918 28278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:01.418977 28278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:01.423979 28278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:01.424110 28278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:01.424132 28278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:01.424150 28278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:01.424198 28278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:01.430552 28278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:01.431867 28278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:01.433037 28278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:28:01.444978 28278 server_base.cc:1034] running on GCE node
W20260806 12:28:01.444906 28284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:01.444914 28285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:01.444938 28288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:01.445868 28278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:01.447347 28278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:01.448606 28278 hybrid_clock.cc:648] HybridClock initialized: now 1786019281448543 us; error 95 us; skew 500 ppm
Aug 06 12:28:01 dist-test-slave-lgr5 krb5kdc[28256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:01.452234 28278 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 12:28:01.453455 28278 webserver.cc:460] Webserver started at http://127.0.0.1:38521/ using document root <none> and password file <none>
I20260806 12:28:01.454114 28278 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:01.454193 28278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:01.454413 28278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:01.456947 28278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "17475fbb85394e39adae53ca425a1b74"
format_stamp: "Formatted at 2026-08-06 12:28:01 on dist-test-slave-lgr5"
server_key: "9c76a928f3ea36dd63e2808ad36516e5"
server_key_iv: "3c9dc0b039a15bdb272714865ea890d0"
server_key_version: "encryptionkey@0"
I20260806 12:28:01.457650 28278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "17475fbb85394e39adae53ca425a1b74"
format_stamp: "Formatted at 2026-08-06 12:28:01 on dist-test-slave-lgr5"
server_key: "9c76a928f3ea36dd63e2808ad36516e5"
server_key_iv: "3c9dc0b039a15bdb272714865ea890d0"
server_key_version: "encryptionkey@0"
I20260806 12:28:01.462551 28278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 12:28:01.465797 28301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:01.467401 28278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 12:28:01.467609 28278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "17475fbb85394e39adae53ca425a1b74"
format_stamp: "Formatted at 2026-08-06 12:28:01 on dist-test-slave-lgr5"
server_key: "9c76a928f3ea36dd63e2808ad36516e5"
server_key_iv: "3c9dc0b039a15bdb272714865ea890d0"
server_key_version: "encryptionkey@0"
I20260806 12:28:01.467751 28278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:01.481494 28278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:01.485955 28278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:01.486228 28278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:01.495410 28278 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42151
I20260806 12:28:01.495432 28383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42151 every 8 connection(s)
I20260806 12:28:01.496800 28278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:01.500811 28385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:01.504562 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 28278
I20260806 12:28:01.504722 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:01.505018 29513 external_mini_cluster.cc:1428] Setting key b65c8302d9c01cf749c8aaa0f94f3ccf
I20260806 12:28:01.508818 28385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74: Bootstrap starting.
I20260806 12:28:01.512598 28385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:01.513707 28385 log.cc:824] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:01.516546 28385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74: No bootstrap required, opened a new log
I20260806 12:28:01.517195 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:01.506785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44641 (local address 127.0.0.1:42151)
0806 12:28:01.507320 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:01.507339 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:01.507383 (+ 44us) server_negotiation.cc:407] Connection header received
0806 12:28:01.508261 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:01.508292 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:01.508890 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:01.509407 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:01.509979 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:01.511460 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:01.512981 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:01.513628 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:01.513778 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:01.514894 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:01.514901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:01.514921 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:01.515397 (+ 476us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:01.515466 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:01.515495 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:01.515615 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:01.515954 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 12:28:01.516170 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":158,"threads_started":1}
I20260806 12:28:01.521086 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:01.521800 28385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17475fbb85394e39adae53ca425a1b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } }
I20260806 12:28:01.522101 28385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:01.522159 28385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17475fbb85394e39adae53ca425a1b74, State: Initialized, Role: FOLLOWER
I20260806 12:28:01.522658 28385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17475fbb85394e39adae53ca425a1b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } }
I20260806 12:28:01.522810 28385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:01.522879 28385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:01.522992 28385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:01.524432 28385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17475fbb85394e39adae53ca425a1b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } }
I20260806 12:28:01.524938 28385 leader_election.cc:302] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17475fbb85394e39adae53ca425a1b74; no voters:
I20260806 12:28:01.525365 28385 leader_election.cc:288] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:01.525530 28399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:01.525799 28399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [term 1 LEADER]: Becoming Leader. State: Replica: 17475fbb85394e39adae53ca425a1b74, State: Running, Role: LEADER
I20260806 12:28:01.526257 28399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17475fbb85394e39adae53ca425a1b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } }
I20260806 12:28:01.526929 28385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:01.527804 28399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17475fbb85394e39adae53ca425a1b74. Latest consensus state: current_term: 1 leader_uuid: "17475fbb85394e39adae53ca425a1b74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17475fbb85394e39adae53ca425a1b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } } }
I20260806 12:28:01.527959 28399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:01.528908 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:01.523498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50393 (local address 127.0.0.1:42151)
0806 12:28:01.523639 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:01.523643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:01.523795 (+ 152us) server_negotiation.cc:407] Connection header received
0806 12:28:01.523916 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:01.523919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:01.523982 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:01.524112 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:01.524390 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:01.525225 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:01.526408 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:01.526780 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:01.526891 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:01.527974 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:01.527979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:01.527982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:01.528000 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:01.528035 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:01.528049 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:01.528174 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:01.528701 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 12:28:01.528753 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 12:28:01.529157 28402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17475fbb85394e39adae53ca425a1b74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17475fbb85394e39adae53ca425a1b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } } }
I20260806 12:28:01.529330 28402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74 [sys.catalog]: This master's current role is: LEADER
W20260806 12:28:01.534035 28426 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:28:01.534199 28426 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:28:01.534332 28406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:01.535629 28406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:01.543212 28406 catalog_manager.cc:1347] Generated new cluster ID: 39280d4476ae40d7a43173267727f54f
I20260806 12:28:01.543453 28406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:01.575783 28406 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:01.577564 28406 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:01.584821 28406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17475fbb85394e39adae53ca425a1b74: Generated new TSK 0
I20260806 12:28:01.585711 28406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:01.597848 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 28278
W20260806 12:28:01.614722 28254 connection.cc:535] client connection to 127.0.0.1:42151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42151): Connection reset by peer (error 104)
2026-08-06T12:28:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1668 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:01 dist-test-slave-lgr5 krb5kdc[28435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:01 dist-test-slave-lgr5 krb5kdc[28435](info): set up 2 sockets
Aug 06 12:28:01 dist-test-slave-lgr5 krb5kdc[28435](info): commencing operation
krb5kdc: starting...
W20260806 12:28:02.816275 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:02 dist-test-slave-lgr5 krb5kdc[28435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 12:28:02.968583 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:03.094187 28455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:03.094524 28455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:03.094576 28455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:03.098863 28455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:03.098953 28455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:03.098973 28455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:03.098996 28455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:03.099011 28455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:03.104772 28455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42321
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:03.105984 28455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:03.107091 28455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:03.116438 28461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:03.116438 28469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:03.117161 28455 server_base.cc:1034] running on GCE node
W20260806 12:28:03.117326 28463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:03.118093 28455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:03.119442 28455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:03.120628 28455 hybrid_clock.cc:648] HybridClock initialized: now 1786019283120606 us; error 39 us; skew 500 ppm
Aug 06 12:28:03 dist-test-slave-lgr5 krb5kdc[28435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:03.123751 28455 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 12:28:03.125029 28455 webserver.cc:460] Webserver started at http://127.0.0.1:40973/ using document root <none> and password file <none>
I20260806 12:28:03.125738 28455 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:03.125830 28455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:03.126084 28455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:03.128947 28455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "83cadcd6a3344de1ada43c552d0bd49f"
format_stamp: "Formatted at 2026-08-06 12:28:03 on dist-test-slave-lgr5"
server_key: "60c11e665e8619eea0751302a4b1c8d6"
server_key_iv: "155b8cf8694549fedec8e612dc00e663"
server_key_version: "encryptionkey@0"
I20260806 12:28:03.129773 28455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "83cadcd6a3344de1ada43c552d0bd49f"
format_stamp: "Formatted at 2026-08-06 12:28:03 on dist-test-slave-lgr5"
server_key: "60c11e665e8619eea0751302a4b1c8d6"
server_key_iv: "155b8cf8694549fedec8e612dc00e663"
server_key_version: "encryptionkey@0"
I20260806 12:28:03.135349 28455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 12:28:03.138490 28481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:03.140182 28455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:28:03.140350 28455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "83cadcd6a3344de1ada43c552d0bd49f"
format_stamp: "Formatted at 2026-08-06 12:28:03 on dist-test-slave-lgr5"
server_key: "60c11e665e8619eea0751302a4b1c8d6"
server_key_iv: "155b8cf8694549fedec8e612dc00e663"
server_key_version: "encryptionkey@0"
I20260806 12:28:03.140488 28455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:03.145841 28455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:03.149470 28455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:03.149869 28455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:03.158278 28455 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42321
I20260806 12:28:03.158279 28565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42321 every 8 connection(s)
I20260806 12:28:03.159384 28455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:03.162735 28566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:03.168054 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 28455
I20260806 12:28:03.168267 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:03.168578 29513 external_mini_cluster.cc:1428] Setting key 4aeb344c74ac33c48a5f39288e9be2fc
I20260806 12:28:03.169613 28566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f: Bootstrap starting.
I20260806 12:28:03.172201 28566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:03.173139 28566 log.cc:824] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:03.174443 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:03.170127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40383 (local address 127.0.0.1:42321)
0806 12:28:03.170618 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:03.170634 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:03.170672 (+ 38us) server_negotiation.cc:407] Connection header received
0806 12:28:03.171350 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:03.171386 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:03.171683 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:03.171986 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:03.172931 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:03.172936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:03.172948 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:03.172958 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:03.173017 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:03.173038 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:03.173127 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:03.173271 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 12:28:03.173462 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":156,"threads_started":1}
I20260806 12:28:03.176046 28566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f: No bootstrap required, opened a new log
I20260806 12:28:03.177922 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:03.181512 28566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cadcd6a3344de1ada43c552d0bd49f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42321 } }
I20260806 12:28:03.181874 28566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:03.181946 28566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83cadcd6a3344de1ada43c552d0bd49f, State: Initialized, Role: FOLLOWER
I20260806 12:28:03.182200 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:03.180465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47755 (local address 127.0.0.1:42321)
0806 12:28:03.180636 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:03.180641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:03.180834 (+ 193us) server_negotiation.cc:407] Connection header received
0806 12:28:03.180888 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:03.180891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:03.180934 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:03.181030 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:03.181716 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:03.181719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:03.181722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:03.181735 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:03.181767 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:03.181781 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:03.181883 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:03.182037 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 12:28:03.182085 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 12:28:03.182459 28566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cadcd6a3344de1ada43c552d0bd49f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42321 } }
I20260806 12:28:03.182629 28566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:03.182706 28566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:03.182812 28566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:03.183931 28566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cadcd6a3344de1ada43c552d0bd49f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42321 } }
I20260806 12:28:03.184316 28566 leader_election.cc:302] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83cadcd6a3344de1ada43c552d0bd49f; no voters:
I20260806 12:28:03.184628 28566 leader_election.cc:288] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:03.184723 28581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:03.184924 28581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [term 1 LEADER]: Becoming Leader. State: Replica: 83cadcd6a3344de1ada43c552d0bd49f, State: Running, Role: LEADER
I20260806 12:28:03.185289 28581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cadcd6a3344de1ada43c552d0bd49f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42321 } }
I20260806 12:28:03.185864 28566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:03.187036 28583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83cadcd6a3344de1ada43c552d0bd49f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cadcd6a3344de1ada43c552d0bd49f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42321 } } }
I20260806 12:28:03.187189 28583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:03.187038 28584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83cadcd6a3344de1ada43c552d0bd49f. Latest consensus state: current_term: 1 leader_uuid: "83cadcd6a3344de1ada43c552d0bd49f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cadcd6a3344de1ada43c552d0bd49f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42321 } } }
I20260806 12:28:03.187274 28584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:03.187655 28600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:03.191679 28600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:03.198410 28600 catalog_manager.cc:1347] Generated new cluster ID: 7fff9c5efdc34553a8b47863bad1f440
I20260806 12:28:03.198724 28600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:03.215937 28600 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:03.216451 28600 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:03.230165 28600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83cadcd6a3344de1ada43c552d0bd49f: Generated new TSK 0
I20260806 12:28:03.231016 28600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:03.243882 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 28455
2026-08-06T12:28:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1659 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:03 dist-test-slave-lgr5 krb5kdc[28618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:03 dist-test-slave-lgr5 krb5kdc[28618](info): set up 2 sockets
Aug 06 12:28:03 dist-test-slave-lgr5 krb5kdc[28618](info): commencing operation
krb5kdc: starting...
W20260806 12:28:04.463063 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:04 dist-test-slave-lgr5 krb5kdc[28618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 12:28:04.611197 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:04.742112 28635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:04.742489 28635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:04.742560 28635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:04.747637 28635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:04.747764 28635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:04.747797 28635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:04.747819 28635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:04.747839 28635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:04.754348 28635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:04.755686 28635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:04.756850 28635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:04.766390 28641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:04.766438 28635 server_base.cc:1034] running on GCE node
W20260806 12:28:04.766448 28642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:04.766556 28644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:04.767477 28635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:04.768957 28635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:04.770187 28635 hybrid_clock.cc:648] HybridClock initialized: now 1786019284770156 us; error 46 us; skew 500 ppm
Aug 06 12:28:04 dist-test-slave-lgr5 krb5kdc[28618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:04.773998 28635 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 12:28:04.775538 28635 webserver.cc:460] Webserver started at http://127.0.0.1:38611/ using document root <none> and password file <none>
I20260806 12:28:04.776288 28635 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:04.776386 28635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:04.776635 28635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:04.779119 28635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "61a01badaee24e4fa6790754a7ed0f52"
format_stamp: "Formatted at 2026-08-06 12:28:04 on dist-test-slave-lgr5"
server_key: "d4fa869a595eaeeb217c540009f8d686"
server_key_iv: "5a9d45ccec89aeab33461f383d494fb0"
server_key_version: "encryptionkey@0"
I20260806 12:28:04.779806 28635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "61a01badaee24e4fa6790754a7ed0f52"
format_stamp: "Formatted at 2026-08-06 12:28:04 on dist-test-slave-lgr5"
server_key: "d4fa869a595eaeeb217c540009f8d686"
server_key_iv: "5a9d45ccec89aeab33461f383d494fb0"
server_key_version: "encryptionkey@0"
I20260806 12:28:04.784322 28635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:28:04.787751 28657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:04.789546 28635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:28:04.789724 28635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "61a01badaee24e4fa6790754a7ed0f52"
format_stamp: "Formatted at 2026-08-06 12:28:04 on dist-test-slave-lgr5"
server_key: "d4fa869a595eaeeb217c540009f8d686"
server_key_iv: "5a9d45ccec89aeab33461f383d494fb0"
server_key_version: "encryptionkey@0"
I20260806 12:28:04.789876 28635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:04.814965 28635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:04.818874 28635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:04.819103 28635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:04.826683 28635 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45287
I20260806 12:28:04.826772 28741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45287 every 8 connection(s)
I20260806 12:28:04.827898 28635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:04.830272 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 28635
I20260806 12:28:04.830420 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:04.830703 29513 external_mini_cluster.cc:1428] Setting key fed0acb0737484c10b567e2a23d2fcac
I20260806 12:28:04.831126 28742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:04.839480 28742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52: Bootstrap starting.
Aug 06 12:28:04 dist-test-slave-lgr5 krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 12:28:04.842586 28742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:04.843613 28742 log.cc:824] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:04.846675 28742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52: No bootstrap required, opened a new log
I20260806 12:28:04.848424 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:04.832143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32793 (local address 127.0.0.1:45287)
0806 12:28:04.832608 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:04.832625 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:04.832663 (+ 38us) server_negotiation.cc:407] Connection header received
0806 12:28:04.833350 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:04.833368 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:04.833738 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:04.834051 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:04.835081 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:04.836501 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:04.838036 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:04.838678 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:04.838840 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:04.842146 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:04.842153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:04.842171 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:04.842213 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:04.844384 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:04.845015 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:04.845022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:04.845029 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:04.845195 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:04.845582 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:04.845586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:04.845588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:04.846224 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:04.846458 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:04.847127 (+ 669us) server_negotiation.cc:299] Negotiation successful
0806 12:28:04.847491 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":138,"threads_started":1}
I20260806 12:28:04.851151 28742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a01badaee24e4fa6790754a7ed0f52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45287 } }
I20260806 12:28:04.851451 28742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:04.851516 28742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61a01badaee24e4fa6790754a7ed0f52, State: Initialized, Role: FOLLOWER
I20260806 12:28:04.851788 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:04.852062 28742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a01badaee24e4fa6790754a7ed0f52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45287 } }
I20260806 12:28:04.852262 28742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:04.852334 28742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:04.852443 28742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:04.853897 28742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a01badaee24e4fa6790754a7ed0f52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45287 } }
I20260806 12:28:04.854336 28742 leader_election.cc:302] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61a01badaee24e4fa6790754a7ed0f52; no voters:
I20260806 12:28:04.854820 28742 leader_election.cc:288] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:04.854902 28754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:04.855163 28754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [term 1 LEADER]: Becoming Leader. State: Replica: 61a01badaee24e4fa6790754a7ed0f52, State: Running, Role: LEADER
I20260806 12:28:04.855546 28754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a01badaee24e4fa6790754a7ed0f52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45287 } }
I20260806 12:28:04.856446 28742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:04.856994 28760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61a01badaee24e4fa6790754a7ed0f52. Latest consensus state: current_term: 1 leader_uuid: "61a01badaee24e4fa6790754a7ed0f52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a01badaee24e4fa6790754a7ed0f52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45287 } } }
I20260806 12:28:04.857141 28760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:04.857872 28757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61a01badaee24e4fa6790754a7ed0f52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a01badaee24e4fa6790754a7ed0f52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45287 } } }
I20260806 12:28:04.858045 28757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:04.858435 28767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:04.862746 28767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:04.865482 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:04.854379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55637 (local address 127.0.0.1:45287)
0806 12:28:04.854530 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:04.854535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:04.854661 (+ 126us) server_negotiation.cc:407] Connection header received
0806 12:28:04.854765 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:04.854768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:04.854824 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:04.854912 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:04.855859 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:04.856693 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:04.857912 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:04.858242 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:04.858361 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:04.860765 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:04.860770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:04.860773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:04.860805 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:04.863214 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:04.863799 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:04.863804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:04.863806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:04.863913 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:04.864307 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:04.864311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:04.864313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:04.864564 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:04.864698 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:04.865010 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 12:28:04.865154 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 12:28:04.869367 28767 catalog_manager.cc:1347] Generated new cluster ID: 19e70d0eaff14eb5935e0d6291086ddd
I20260806 12:28:04.869465 28767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:04.893103 28767 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:04.894436 28767 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:04.901618 28767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61a01badaee24e4fa6790754a7ed0f52: Generated new TSK 0
I20260806 12:28:04.902747 28767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:04.907619 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 28635
2026-08-06T12:28:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1648 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:04 dist-test-slave-lgr5 krb5kdc[28791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:04 dist-test-slave-lgr5 krb5kdc[28791](info): set up 2 sockets
Aug 06 12:28:04 dist-test-slave-lgr5 krb5kdc[28791](info): commencing operation
krb5kdc: starting...
W20260806 12:28:06.106545 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:06 dist-test-slave-lgr5 krb5kdc[28791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 12:28:06.256925 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:06.380119 28809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:06.381043 28809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:06.381141 28809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:06.386003 28809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:06.386112 28809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:06.386143 28809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:06.386164 28809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:06.386186 28809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:06.392025 28809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:06.393200 28809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:06.394245 28809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:06.402719 28819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:06.402721 28817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:06.402809 28816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:06.402825 28809 server_base.cc:1034] running on GCE node
I20260806 12:28:06.403563 28809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:06.404667 28809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:06.405833 28809 hybrid_clock.cc:648] HybridClock initialized: now 1786019286405808 us; error 37 us; skew 500 ppm
Aug 06 12:28:06 dist-test-slave-lgr5 krb5kdc[28791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:06.408655 28809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 12:28:06.409600 28809 webserver.cc:460] Webserver started at http://127.0.0.1:34669/ using document root <none> and password file <none>
I20260806 12:28:06.410152 28809 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:06.410221 28809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:06.410441 28809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:06.412601 28809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "e9f8589317e14fb1b4980e81204eb1c2"
format_stamp: "Formatted at 2026-08-06 12:28:06 on dist-test-slave-lgr5"
server_key: "d222301eead67048d3392f6fcf5c1e29"
server_key_iv: "df2b679229e57629f061c2be067eb847"
server_key_version: "encryptionkey@0"
I20260806 12:28:06.413188 28809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "e9f8589317e14fb1b4980e81204eb1c2"
format_stamp: "Formatted at 2026-08-06 12:28:06 on dist-test-slave-lgr5"
server_key: "d222301eead67048d3392f6fcf5c1e29"
server_key_iv: "df2b679229e57629f061c2be067eb847"
server_key_version: "encryptionkey@0"
I20260806 12:28:06.417336 28809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 12:28:06.420198 28832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:06.421588 28809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:28:06.421738 28809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "e9f8589317e14fb1b4980e81204eb1c2"
format_stamp: "Formatted at 2026-08-06 12:28:06 on dist-test-slave-lgr5"
server_key: "d222301eead67048d3392f6fcf5c1e29"
server_key_iv: "df2b679229e57629f061c2be067eb847"
server_key_version: "encryptionkey@0"
I20260806 12:28:06.421859 28809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:06.446964 28809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:06.450372 28809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:06.450588 28809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:06.458127 28809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41819
I20260806 12:28:06.458194 28908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41819 every 8 connection(s)
I20260806 12:28:06.459168 28809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:06.462234 28911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:06.465385 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 28809
I20260806 12:28:06.465534 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:06.465785 29513 external_mini_cluster.cc:1428] Setting key f8081a34c0fc5a62f9130545e5763403
I20260806 12:28:06.468712 28911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2: Bootstrap starting.
I20260806 12:28:06.471138 28911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:06.471979 28911 log.cc:824] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:06.474435 28911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2: No bootstrap required, opened a new log
I20260806 12:28:06.475522 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:06.467222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37675 (local address 127.0.0.1:41819)
0806 12:28:06.467751 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:06.467767 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:06.467807 (+ 40us) server_negotiation.cc:407] Connection header received
0806 12:28:06.468513 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:06.468532 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:06.468911 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:06.469236 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:06.469668 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:06.470713 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:06.471683 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:06.472272 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:06.472446 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:06.473326 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:06.473333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:06.473351 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:06.473855 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:06.473944 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:06.473983 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:06.474085 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:06.474234 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 12:28:06.474461 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":168,"threads_started":1}
I20260806 12:28:06.479136 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:06.479205 28911 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f8589317e14fb1b4980e81204eb1c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260806 12:28:06.479530 28911 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:06.479604 28911 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f8589317e14fb1b4980e81204eb1c2, State: Initialized, Role: FOLLOWER
I20260806 12:28:06.480247 28911 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f8589317e14fb1b4980e81204eb1c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260806 12:28:06.480475 28911 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:06.480592 28911 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:06.480711 28911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:06.482311 28911 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f8589317e14fb1b4980e81204eb1c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260806 12:28:06.482770 28911 leader_election.cc:302] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9f8589317e14fb1b4980e81204eb1c2; no voters:
I20260806 12:28:06.483136 28911 leader_election.cc:288] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:06.483234 28928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:06.483425 28928 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [term 1 LEADER]: Becoming Leader. State: Replica: e9f8589317e14fb1b4980e81204eb1c2, State: Running, Role: LEADER
I20260806 12:28:06.484130 28928 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f8589317e14fb1b4980e81204eb1c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260806 12:28:06.484980 28911 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:06.486053 28930 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9f8589317e14fb1b4980e81204eb1c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f8589317e14fb1b4980e81204eb1c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } } }
I20260806 12:28:06.486200 28930 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:06.486443 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:06.481983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52857 (local address 127.0.0.1:41819)
0806 12:28:06.482133 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:06.482138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:06.482154 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:28:06.482199 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:06.482202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:06.482250 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:06.482316 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:06.482623 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:06.483266 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:06.484375 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:06.484660 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:06.484756 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:06.485626 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:06.485631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:06.485634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:06.485648 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:06.485683 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:06.485696 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:06.486190 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:06.486230 (+ 40us) server_negotiation.cc:299] Negotiation successful
0806 12:28:06.486296 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 12:28:06.486584 28929 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f8589317e14fb1b4980e81204eb1c2. Latest consensus state: current_term: 1 leader_uuid: "e9f8589317e14fb1b4980e81204eb1c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f8589317e14fb1b4980e81204eb1c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } } }
I20260806 12:28:06.486689 28929 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2 [sys.catalog]: This master's current role is: LEADER
W20260806 12:28:06.490301 28955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:28:06.490446 28955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:28:06.490536 28947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:06.491314 28947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:06.498221 28947 catalog_manager.cc:1347] Generated new cluster ID: 6d6b7db9dbe743ecb848b53097498395
I20260806 12:28:06.498308 28947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:06.523787 28947 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:06.525295 28947 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:06.530745 28947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9f8589317e14fb1b4980e81204eb1c2: Generated new TSK 0
I20260806 12:28:06.531685 28947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:06.544867 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 28809
2026-08-06T12:28:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1634 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:06 dist-test-slave-lgr5 krb5kdc[28961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:06 dist-test-slave-lgr5 krb5kdc[28961](info): set up 2 sockets
Aug 06 12:28:06 dist-test-slave-lgr5 krb5kdc[28961](info): commencing operation
krb5kdc: starting...
W20260806 12:28:07.794320 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:07 dist-test-slave-lgr5 krb5kdc[28961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 12:28:07.940318 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:08.079490 28982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:08.079831 28982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:08.079877 28982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:08.084527 28982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:08.084646 28982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:08.084672 28982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:08.084693 28982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:08.084712 28982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:08.090919 28982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:08.092314 28982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:08.093761 28982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:08.104249 28998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:08.104249 28991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:08.104250 28993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:08.105037 28982 server_base.cc:1034] running on GCE node
I20260806 12:28:08.105734 28982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:08.107128 28982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:08.108326 28982 hybrid_clock.cc:648] HybridClock initialized: now 1786019288108295 us; error 49 us; skew 500 ppm
Aug 06 12:28:08 dist-test-slave-lgr5 krb5kdc[28961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:08.111837 28982 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 12:28:08.113044 28982 webserver.cc:460] Webserver started at http://127.0.0.1:42099/ using document root <none> and password file <none>
I20260806 12:28:08.113719 28982 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:08.113806 28982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:08.114061 28982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:08.117051 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "36c97a0c95d94b19b4ff72ace25515af"
format_stamp: "Formatted at 2026-08-06 12:28:08 on dist-test-slave-lgr5"
server_key: "2ca585a1bbf546c23a955466810d5ef0"
server_key_iv: "45df663efaf4f863fec1d204518dc77e"
server_key_version: "encryptionkey@0"
I20260806 12:28:08.118247 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "36c97a0c95d94b19b4ff72ace25515af"
format_stamp: "Formatted at 2026-08-06 12:28:08 on dist-test-slave-lgr5"
server_key: "2ca585a1bbf546c23a955466810d5ef0"
server_key_iv: "45df663efaf4f863fec1d204518dc77e"
server_key_version: "encryptionkey@0"
I20260806 12:28:08.124358 28982 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 12:28:08.127718 29007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:08.129516 28982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:28:08.129689 28982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "36c97a0c95d94b19b4ff72ace25515af"
format_stamp: "Formatted at 2026-08-06 12:28:08 on dist-test-slave-lgr5"
server_key: "2ca585a1bbf546c23a955466810d5ef0"
server_key_iv: "45df663efaf4f863fec1d204518dc77e"
server_key_version: "encryptionkey@0"
I20260806 12:28:08.129833 28982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:08.135861 28982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:08.140703 28982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:08.140995 28982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:08.150261 28982 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38813
I20260806 12:28:08.150293 29089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38813 every 8 connection(s)
I20260806 12:28:08.151473 28982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:08.154987 29091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:08.160259 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 28982
I20260806 12:28:08.160409 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:08.160724 29513 external_mini_cluster.cc:1428] Setting key 068faf8b91df6ce810bf7e4cab2774da
I20260806 12:28:08.162520 29091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af: Bootstrap starting.
I20260806 12:28:08.166077 29091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:08.167248 29091 log.cc:824] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:08.167965 29095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:08.162414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39693 (local address 127.0.0.1:38813)
0806 12:28:08.163081 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:08.163102 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:08.163150 (+ 48us) server_negotiation.cc:407] Connection header received
0806 12:28:08.164233 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:08.164266 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:08.164702 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:08.165190 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:08.166076 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:08.166082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:08.166100 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:08.166115 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:08.166188 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:08.166222 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:08.166352 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:08.166571 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 12:28:08.166775 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":215,"threads_started":1}
I20260806 12:28:08.170123 29091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af: No bootstrap required, opened a new log
I20260806 12:28:08.171790 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:08.175097 29091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c97a0c95d94b19b4ff72ace25515af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260806 12:28:08.175432 29091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:08.175499 29091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c97a0c95d94b19b4ff72ace25515af, State: Initialized, Role: FOLLOWER
I20260806 12:28:08.176183 29095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:08.174283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43973 (local address 127.0.0.1:38813)
0806 12:28:08.174448 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:08.174453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:08.174576 (+ 123us) server_negotiation.cc:407] Connection header received
0806 12:28:08.174708 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:08.174711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:08.174759 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:08.174863 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:08.175721 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:08.175725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:08.175728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:08.175746 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:08.175786 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:08.175801 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:08.175906 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:08.175962 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 12:28:08.176010 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 12:28:08.176133 29091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c97a0c95d94b19b4ff72ace25515af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260806 12:28:08.176324 29091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:08.176406 29091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:08.176522 29091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:08.178036 29091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c97a0c95d94b19b4ff72ace25515af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260806 12:28:08.178472 29091 leader_election.cc:302] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36c97a0c95d94b19b4ff72ace25515af; no voters:
I20260806 12:28:08.178865 29091 leader_election.cc:288] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:08.179059 29107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:08.179301 29107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [term 1 LEADER]: Becoming Leader. State: Replica: 36c97a0c95d94b19b4ff72ace25515af, State: Running, Role: LEADER
I20260806 12:28:08.179657 29107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c97a0c95d94b19b4ff72ace25515af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260806 12:28:08.180353 29091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:08.181572 29112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36c97a0c95d94b19b4ff72ace25515af. Latest consensus state: current_term: 1 leader_uuid: "36c97a0c95d94b19b4ff72ace25515af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c97a0c95d94b19b4ff72ace25515af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } } }
I20260806 12:28:08.181741 29112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:08.182211 29119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:08.182765 29108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36c97a0c95d94b19b4ff72ace25515af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c97a0c95d94b19b4ff72ace25515af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } } }
I20260806 12:28:08.182919 29108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:08.186408 29119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:08.193202 29119 catalog_manager.cc:1347] Generated new cluster ID: d79a806bfb9f42cd9742a46ce884d0df
I20260806 12:28:08.193387 29119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:08.219583 29119 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:08.220059 29119 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:08.231195 29119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36c97a0c95d94b19b4ff72ace25515af: Generated new TSK 0
I20260806 12:28:08.232381 29119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:08.242547 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 28982
2026-08-06T12:28:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1700 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:08 dist-test-slave-lgr5 krb5kdc[29151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:08 dist-test-slave-lgr5 krb5kdc[29151](info): set up 2 sockets
Aug 06 12:28:08 dist-test-slave-lgr5 krb5kdc[29151](info): commencing operation
krb5kdc: starting...
W20260806 12:28:09.429039 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:09 dist-test-slave-lgr5 krb5kdc[29151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 12:28:09.581476 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:09.709445 29173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:09.709741 29173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:09.709784 29173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:09.713874 29173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:09.713954 29173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:09.713972 29173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:09.713989 29173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:09.714002 29173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:09.719775 29173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:09.721020 29173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:09.722084 29173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:09.731863 29180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:09.731951 29179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:09.731987 29182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:09.732033 29173 server_base.cc:1034] running on GCE node
I20260806 12:28:09.732884 29173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:09.734263 29173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:09.735458 29173 hybrid_clock.cc:648] HybridClock initialized: now 1786019289735442 us; error 37 us; skew 500 ppm
Aug 06 12:28:09 dist-test-slave-lgr5 krb5kdc[29151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:09.738817 29173 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 12:28:09.739967 29173 webserver.cc:460] Webserver started at http://127.0.0.1:43147/ using document root <none> and password file <none>
I20260806 12:28:09.740651 29173 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:09.740731 29173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:09.740968 29173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:09.743232 29173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "aa30856a980f4b549856c8146e998cba"
format_stamp: "Formatted at 2026-08-06 12:28:09 on dist-test-slave-lgr5"
server_key: "a6f5f8787008c7df8865fa2e30916207"
server_key_iv: "65efb3834eb7451d839d1f1f2f97762c"
server_key_version: "encryptionkey@0"
I20260806 12:28:09.743818 29173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "aa30856a980f4b549856c8146e998cba"
format_stamp: "Formatted at 2026-08-06 12:28:09 on dist-test-slave-lgr5"
server_key: "a6f5f8787008c7df8865fa2e30916207"
server_key_iv: "65efb3834eb7451d839d1f1f2f97762c"
server_key_version: "encryptionkey@0"
I20260806 12:28:09.748158 29173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260806 12:28:09.751021 29195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:09.752410 29173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 12:28:09.752578 29173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "aa30856a980f4b549856c8146e998cba"
format_stamp: "Formatted at 2026-08-06 12:28:09 on dist-test-slave-lgr5"
server_key: "a6f5f8787008c7df8865fa2e30916207"
server_key_iv: "65efb3834eb7451d839d1f1f2f97762c"
server_key_version: "encryptionkey@0"
I20260806 12:28:09.752703 29173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:09.770741 29173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:09.774180 29173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:09.774395 29173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:09.781840 29173 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37347
I20260806 12:28:09.781864 29286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37347 every 8 connection(s)
I20260806 12:28:09.782877 29173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:09.786298 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:09.790607 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 29173
I20260806 12:28:09.790776 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:09.791090 29513 external_mini_cluster.cc:1428] Setting key 8cdfd2525a22edf5a24fd0041abb482d
I20260806 12:28:09.793398 29287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba: Bootstrap starting.
I20260806 12:28:09.796224 29287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:09.797068 29287 log.cc:824] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:09.799546 29287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba: No bootstrap required, opened a new log
Aug 06 12:28:09 dist-test-slave-lgr5 krb5kdc[29151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 12:28:09.803792 29287 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa30856a980f4b549856c8146e998cba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } }
I20260806 12:28:09.804138 29287 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:09.804201 29287 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa30856a980f4b549856c8146e998cba, State: Initialized, Role: FOLLOWER
I20260806 12:28:09.804726 29287 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa30856a980f4b549856c8146e998cba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } }
I20260806 12:28:09.804867 29287 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:09.804941 29287 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:09.805037 29287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:09.806212 29287 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa30856a980f4b549856c8146e998cba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } }
I20260806 12:28:09.806562 29287 leader_election.cc:302] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa30856a980f4b549856c8146e998cba; no voters:
I20260806 12:28:09.806617 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:09.792738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:37563 (local address 127.0.0.1:37347)
0806 12:28:09.793262 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:09.793280 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:09.793320 (+ 40us) server_negotiation.cc:407] Connection header received
0806 12:28:09.794058 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:09.794077 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:09.794470 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:09.794773 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:09.795744 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:09.796950 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:09.798020 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:09.798515 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:09.798647 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:09.801670 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:09.801676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:09.801692 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:09.801721 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:09.803379 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:09.803958 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:09.803963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:09.803968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:09.804114 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:09.804478 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:09.804482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:09.804483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:09.804986 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:09.805224 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:09.805499 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 12:28:09.805731 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":154,"threads_started":1}
I20260806 12:28:09.806885 29287 leader_election.cc:288] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:09.806974 29295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:09.807176 29295 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [term 1 LEADER]: Becoming Leader. State: Replica: aa30856a980f4b549856c8146e998cba, State: Running, Role: LEADER
I20260806 12:28:09.807554 29295 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa30856a980f4b549856c8146e998cba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } }
I20260806 12:28:09.809340 29287 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:09.809316 29296 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa30856a980f4b549856c8146e998cba. Latest consensus state: current_term: 1 leader_uuid: "aa30856a980f4b549856c8146e998cba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa30856a980f4b549856c8146e998cba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } } }
I20260806 12:28:09.809520 29296 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:09.809926 29301 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa30856a980f4b549856c8146e998cba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa30856a980f4b549856c8146e998cba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37347 } } }
I20260806 12:28:09.810032 29301 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:09.810020 29305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:09.810956 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:09.814424 29305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:09.822139 29305 catalog_manager.cc:1347] Generated new cluster ID: 1847afe6adeb4c9a80d420b9e192a9f0
I20260806 12:28:09.822240 29305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:09.823523 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:09.813458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:45773 (local address 127.0.0.1:37347)
0806 12:28:09.813585 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:09.813590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:09.813838 (+ 248us) server_negotiation.cc:407] Connection header received
0806 12:28:09.813984 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:09.813988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:09.814042 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:09.814129 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:09.815110 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:09.815874 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:09.817340 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:09.817700 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:09.817821 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:09.820066 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:09.820069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:09.820071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:09.820131 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:09.821545 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:09.822137 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:09.822140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:09.822142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:09.822255 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:09.822635 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:09.822639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:09.822640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:09.822827 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:09.822971 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:09.823256 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 12:28:09.823362 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 12:28:09.845749 29305 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:09.846786 29305 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:09.852547 29305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa30856a980f4b549856c8146e998cba: Generated new TSK 0
I20260806 12:28:09.853520 29305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:09.865767 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 29173
W20260806 12:28:09.884773 29148 connection.cc:535] client connection to 127.0.0.1:37347 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37347): Connection reset by peer (error 104)
2026-08-06T12:28:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1626 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:09 dist-test-slave-lgr5 krb5kdc[29337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:09 dist-test-slave-lgr5 krb5kdc[29337](info): set up 2 sockets
Aug 06 12:28:09 dist-test-slave-lgr5 krb5kdc[29337](info): commencing operation
krb5kdc: starting...
W20260806 12:28:11.060236 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:11 dist-test-slave-lgr5 krb5kdc[29337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 12:28:11.203771 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:11.332438 29356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:11.332782 29356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:11.332849 29356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:11.337198 29356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:11.337291 29356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:11.337320 29356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:11.337342 29356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:11.337363 29356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:11.343262 29356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:11.344575 29356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:11.345805 29356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:11.355545 29364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:11.355545 29363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:11.355546 29367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:11.355726 29356 server_base.cc:1034] running on GCE node
I20260806 12:28:11.356679 29356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:11.358009 29356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:11.359222 29356 hybrid_clock.cc:648] HybridClock initialized: now 1786019291359187 us; error 52 us; skew 500 ppm
Aug 06 12:28:11 dist-test-slave-lgr5 krb5kdc[29337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:11.362576 29356 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 12:28:11.363771 29356 webserver.cc:460] Webserver started at http://127.0.0.1:34857/ using document root <none> and password file <none>
I20260806 12:28:11.364480 29356 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:11.364557 29356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:11.364778 29356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:11.367259 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "f9096c01122442c89f9ffa4f92dacce7"
format_stamp: "Formatted at 2026-08-06 12:28:11 on dist-test-slave-lgr5"
server_key: "9ba0e66ff5cba5e02ee5a1da9972b1a8"
server_key_iv: "d58531f44ea64a69f453dfcda4595def"
server_key_version: "encryptionkey@0"
I20260806 12:28:11.367944 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "f9096c01122442c89f9ffa4f92dacce7"
format_stamp: "Formatted at 2026-08-06 12:28:11 on dist-test-slave-lgr5"
server_key: "9ba0e66ff5cba5e02ee5a1da9972b1a8"
server_key_iv: "d58531f44ea64a69f453dfcda4595def"
server_key_version: "encryptionkey@0"
I20260806 12:28:11.373109 29356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 12:28:11.377292 29378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:11.379478 29356 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 12:28:11.379673 29356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "f9096c01122442c89f9ffa4f92dacce7"
format_stamp: "Formatted at 2026-08-06 12:28:11 on dist-test-slave-lgr5"
server_key: "9ba0e66ff5cba5e02ee5a1da9972b1a8"
server_key_iv: "d58531f44ea64a69f453dfcda4595def"
server_key_version: "encryptionkey@0"
I20260806 12:28:11.379863 29356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:11.410643 29356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:11.414286 29356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:11.414631 29356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:11.423271 29356 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40381
I20260806 12:28:11.423295 29459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40381 every 8 connection(s)
I20260806 12:28:11.424474 29356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:11.428551 29461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:11.433601 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 29356
I20260806 12:28:11.433746 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:11.434062 29513 external_mini_cluster.cc:1428] Setting key b18acc45dfe18fca04cf8bf0b3589b82
I20260806 12:28:11.436162 29461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7: Bootstrap starting.
I20260806 12:28:11.439525 29461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:11.440668 29461 log.cc:824] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:11.444283 29461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7: No bootstrap required, opened a new log
I20260806 12:28:11.447973 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:11.436186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:43567 (local address 127.0.0.1:40381)
0806 12:28:11.436790 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:11.436824 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:11.436875 (+ 51us) server_negotiation.cc:407] Connection header received
0806 12:28:11.437904 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:11.437934 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:11.438469 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:11.438906 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:11.439427 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:11.441045 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:11.442371 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:11.443257 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:11.443610 (+ 353us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:11.444835 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:11.444844 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:11.444872 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:11.445546 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:11.445647 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:11.445687 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:11.445896 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:11.446383 (+ 487us) server_negotiation.cc:299] Negotiation successful
0806 12:28:11.446596 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":158,"threads_started":1}
I20260806 12:28:11.449725 29461 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9096c01122442c89f9ffa4f92dacce7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40381 } }
I20260806 12:28:11.450093 29461 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:11.450148 29461 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9096c01122442c89f9ffa4f92dacce7, State: Initialized, Role: FOLLOWER
I20260806 12:28:11.450662 29461 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9096c01122442c89f9ffa4f92dacce7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40381 } }
I20260806 12:28:11.450932 29461 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:11.451006 29461 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:11.451102 29461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:11.452235 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:11.452549 29461 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9096c01122442c89f9ffa4f92dacce7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40381 } }
I20260806 12:28:11.453111 29461 leader_election.cc:302] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9096c01122442c89f9ffa4f92dacce7; no voters:
I20260806 12:28:11.453516 29461 leader_election.cc:288] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:11.453606 29467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:11.453819 29467 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [term 1 LEADER]: Becoming Leader. State: Replica: f9096c01122442c89f9ffa4f92dacce7, State: Running, Role: LEADER
I20260806 12:28:11.454205 29467 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9096c01122442c89f9ffa4f92dacce7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40381 } }
I20260806 12:28:11.455157 29461 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:11.456177 29474 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9096c01122442c89f9ffa4f92dacce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9096c01122442c89f9ffa4f92dacce7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40381 } } }
I20260806 12:28:11.456334 29474 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:11.456686 29475 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9096c01122442c89f9ffa4f92dacce7. Latest consensus state: current_term: 1 leader_uuid: "f9096c01122442c89f9ffa4f92dacce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9096c01122442c89f9ffa4f92dacce7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40381 } } }
I20260806 12:28:11.456775 29475 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:11.457757 29495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:11.459998 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:11.455088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:59353 (local address 127.0.0.1:40381)
0806 12:28:11.455201 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:11.455206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:11.455221 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:28:11.455389 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:11.455393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:11.455455 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:11.455533 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:11.456003 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:11.456820 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:11.458008 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:11.458352 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:11.458439 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:11.459469 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:11.459473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:11.459475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:11.459492 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:11.459522 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:11.459546 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:11.459644 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:11.459804 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 12:28:11.459848 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 12:28:11.461519 29495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:11.468520 29495 catalog_manager.cc:1347] Generated new cluster ID: bd0bee95622b491186f029af88f87668
I20260806 12:28:11.468631 29495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:11.480405 29495 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:11.482961 29495 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:11.491688 29495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9096c01122442c89f9ffa4f92dacce7: Generated new TSK 0
I20260806 12:28:11.492739 29495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:11.503237 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 29356
W20260806 12:28:11.527305 29335 connection.cc:535] client connection to 127.0.0.1:40381 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40381): Connection reset by peer (error 104)
2026-08-06T12:28:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1642 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:11 dist-test-slave-lgr5 krb5kdc[29509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:11 dist-test-slave-lgr5 krb5kdc[29509](info): set up 2 sockets
Aug 06 12:28:11 dist-test-slave-lgr5 krb5kdc[29509](info): commencing operation
krb5kdc: starting...
W20260806 12:28:12.685902 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:12 dist-test-slave-lgr5 krb5kdc[29509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 12:28:12.835662 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:12.962168 29532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:12.962517 29532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:12.962567 29532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:12.966893 29532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:12.966991 29532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:12.967010 29532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:12.967026 29532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:12.967041 29532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:12.973484 29532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:12.974799 29532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:12.976478 29532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:12.988327 29542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:12.988440 29540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:12.988327 29539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:12.990895 29532 server_base.cc:1034] running on GCE node
I20260806 12:28:12.991715 29532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:12.993474 29532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:12.994685 29532 hybrid_clock.cc:648] HybridClock initialized: now 1786019292994646 us; error 67 us; skew 500 ppm
Aug 06 12:28:12 dist-test-slave-lgr5 krb5kdc[29509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:12.999193 29532 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 12:28:13.002717 29532 webserver.cc:460] Webserver started at http://127.0.0.1:36283/ using document root <none> and password file <none>
I20260806 12:28:13.004824 29532 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:13.005005 29532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:13.005384 29532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:13.009172 29532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "0a7fc5971f4a450db4199c8f26e9149d"
format_stamp: "Formatted at 2026-08-06 12:28:13 on dist-test-slave-lgr5"
server_key: "eb202e812137d2cca4b41ad71b8c5c2e"
server_key_iv: "02303d453fd2cd2bc21930a948c73553"
server_key_version: "encryptionkey@0"
I20260806 12:28:13.009943 29532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "0a7fc5971f4a450db4199c8f26e9149d"
format_stamp: "Formatted at 2026-08-06 12:28:13 on dist-test-slave-lgr5"
server_key: "eb202e812137d2cca4b41ad71b8c5c2e"
server_key_iv: "02303d453fd2cd2bc21930a948c73553"
server_key_version: "encryptionkey@0"
I20260806 12:28:13.018014 29532 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260806 12:28:13.021554 29556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:13.023355 29532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:28:13.023605 29532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "0a7fc5971f4a450db4199c8f26e9149d"
format_stamp: "Formatted at 2026-08-06 12:28:13 on dist-test-slave-lgr5"
server_key: "eb202e812137d2cca4b41ad71b8c5c2e"
server_key_iv: "02303d453fd2cd2bc21930a948c73553"
server_key_version: "encryptionkey@0"
I20260806 12:28:13.023769 29532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:13.029893 29532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:13.034144 29532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:13.034381 29532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:13.045142 29532 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41873
I20260806 12:28:13.045147 29643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41873 every 8 connection(s)
I20260806 12:28:13.046413 29532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:13.046918 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 29532
I20260806 12:28:13.047055 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:13.047407 29513 external_mini_cluster.cc:1428] Setting key c10a04ab0b1df8e68e9e30fd31a67604
I20260806 12:28:13.050072 29644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:13.053834 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:13.049011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:37299 (local address 127.0.0.1:41873)
0806 12:28:13.049535 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:13.049560 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:13.049731 (+ 171us) server_negotiation.cc:407] Connection header received
0806 12:28:13.050635 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:13.050658 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:13.051016 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:13.051402 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:13.052329 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:13.052334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:13.052348 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:13.052360 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:13.052434 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:13.052457 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:13.052609 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:13.052689 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 12:28:13.052880 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":139,"threads_started":1}
I20260806 12:28:13.057533 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:13.058099 29644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d: Bootstrap starting.
I20260806 12:28:13.061242 29644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:13.062114 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:13.060185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:39617 (local address 127.0.0.1:41873)
0806 12:28:13.060314 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:13.060318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:13.060603 (+ 285us) server_negotiation.cc:407] Connection header received
0806 12:28:13.060742 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:13.060745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:13.060782 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:13.060860 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:13.061720 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:13.061723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:13.061725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:13.061738 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:13.061767 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:13.061778 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:13.061864 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:13.061963 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 12:28:13.062003 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 12:28:13.062321 29644 log.cc:824] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:13.064697 29644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d: No bootstrap required, opened a new log
I20260806 12:28:13.069013 29644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7fc5971f4a450db4199c8f26e9149d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } }
I20260806 12:28:13.069316 29644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:13.069356 29644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7fc5971f4a450db4199c8f26e9149d, State: Initialized, Role: FOLLOWER
I20260806 12:28:13.069809 29644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7fc5971f4a450db4199c8f26e9149d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } }
I20260806 12:28:13.069972 29644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:13.070025 29644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:13.070104 29644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:13.071173 29644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7fc5971f4a450db4199c8f26e9149d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } }
I20260806 12:28:13.071554 29644 leader_election.cc:302] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a7fc5971f4a450db4199c8f26e9149d; no voters:
I20260806 12:28:13.071907 29644 leader_election.cc:288] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:13.072130 29662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:13.072409 29662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [term 1 LEADER]: Becoming Leader. State: Replica: 0a7fc5971f4a450db4199c8f26e9149d, State: Running, Role: LEADER
I20260806 12:28:13.072862 29662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7fc5971f4a450db4199c8f26e9149d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } }
I20260806 12:28:13.073474 29644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:13.074788 29663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a7fc5971f4a450db4199c8f26e9149d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7fc5971f4a450db4199c8f26e9149d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } } }
I20260806 12:28:13.074937 29663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:13.075153 29667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a7fc5971f4a450db4199c8f26e9149d. Latest consensus state: current_term: 1 leader_uuid: "0a7fc5971f4a450db4199c8f26e9149d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7fc5971f4a450db4199c8f26e9149d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41873 } } }
I20260806 12:28:13.075321 29667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:13.075402 29682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:13.078548 29682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:13.084475 29682 catalog_manager.cc:1347] Generated new cluster ID: d6a9ef1d64f94b4087834953582d4848
I20260806 12:28:13.084604 29682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:13.098352 29682 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:13.098719 29682 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:13.103228 29682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a7fc5971f4a450db4199c8f26e9149d: Generated new TSK 0
I20260806 12:28:13.104207 29682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:13.117092 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 29532
2026-08-06T12:28:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1607 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:13 dist-test-slave-lgr5 krb5kdc[29698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:13 dist-test-slave-lgr5 krb5kdc[29698](info): set up 2 sockets
Aug 06 12:28:13 dist-test-slave-lgr5 krb5kdc[29698](info): commencing operation
krb5kdc: starting...
W20260806 12:28:14.312572 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:14 dist-test-slave-lgr5 krb5kdc[29698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 12:28:14.458637 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:14.593866 29722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:14.594219 29722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:14.594286 29722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:14.598824 29722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:14.598922 29722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:14.598950 29722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:14.598971 29722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:14.598991 29722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:14.605108 29722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:14.606428 29722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:14.607666 29722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:14.617859 29731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:14.617859 29730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:14.617872 29734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:14.618619 29722 server_base.cc:1034] running on GCE node
I20260806 12:28:14.619262 29722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:14.620687 29722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:14.621884 29722 hybrid_clock.cc:648] HybridClock initialized: now 1786019294621868 us; error 34 us; skew 500 ppm
Aug 06 12:28:14 dist-test-slave-lgr5 krb5kdc[29698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:14.625203 29722 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 12:28:14.626607 29722 webserver.cc:460] Webserver started at http://127.0.0.1:39597/ using document root <none> and password file <none>
I20260806 12:28:14.627449 29722 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:14.627540 29722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:14.627785 29722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:14.630604 29722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "2f37118653f144219c7b15a25b97d8b0"
format_stamp: "Formatted at 2026-08-06 12:28:14 on dist-test-slave-lgr5"
server_key: "b7a468ff088a256b3fca7e383ccdd3c3"
server_key_iv: "7fc53d583441cf3400b2228c63bad440"
server_key_version: "encryptionkey@0"
I20260806 12:28:14.631358 29722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "2f37118653f144219c7b15a25b97d8b0"
format_stamp: "Formatted at 2026-08-06 12:28:14 on dist-test-slave-lgr5"
server_key: "b7a468ff088a256b3fca7e383ccdd3c3"
server_key_iv: "7fc53d583441cf3400b2228c63bad440"
server_key_version: "encryptionkey@0"
I20260806 12:28:14.636260 29722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260806 12:28:14.641870 29745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:14.644695 29722 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 12:28:14.644917 29722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "2f37118653f144219c7b15a25b97d8b0"
format_stamp: "Formatted at 2026-08-06 12:28:14 on dist-test-slave-lgr5"
server_key: "b7a468ff088a256b3fca7e383ccdd3c3"
server_key_iv: "7fc53d583441cf3400b2228c63bad440"
server_key_version: "encryptionkey@0"
I20260806 12:28:14.645074 29722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:14.667840 29722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:14.672461 29722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:14.672724 29722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:14.684015 29722 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44625
I20260806 12:28:14.684063 29833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44625 every 8 connection(s)
I20260806 12:28:14.685503 29722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:14.688675 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 29722
I20260806 12:28:14.688871 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:14.689253 29513 external_mini_cluster.cc:1428] Setting key 9d8e42d522a00f4115e0541216e7f9e9
I20260806 12:28:14.689636 29836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:14.698531 29836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0: Bootstrap starting.
Aug 06 12:28:14 dist-test-slave-lgr5 krb5kdc[29698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 12:28:14.701803 29836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:14.702767 29836 log.cc:824] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:14.705931 29836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0: No bootstrap required, opened a new log
I20260806 12:28:14.709105 29840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:14.691044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:57677 (local address 127.0.0.1:44625)
0806 12:28:14.691584 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:14.691603 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:14.691648 (+ 45us) server_negotiation.cc:407] Connection header received
0806 12:28:14.692461 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:14.692484 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:14.692911 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:14.693246 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:14.694330 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:14.695498 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:14.696899 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:14.697653 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:14.697854 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:14.702468 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:14.702477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:14.702508 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:14.702556 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:14.704900 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:14.705592 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:14.705599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:14.705606 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:14.705768 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:14.706145 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:14.706149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:14.706150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:14.706780 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:14.707221 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:14.707486 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 12:28:14.707791 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":173,"threads_started":1}
I20260806 12:28:14.711447 29836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f37118653f144219c7b15a25b97d8b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } }
I20260806 12:28:14.711836 29836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:14.711908 29836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f37118653f144219c7b15a25b97d8b0, State: Initialized, Role: FOLLOWER
I20260806 12:28:14.712498 29836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f37118653f144219c7b15a25b97d8b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } }
I20260806 12:28:14.712685 29836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:14.712752 29836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:14.712870 29836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:14.713349 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:14.714293 29836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f37118653f144219c7b15a25b97d8b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } }
I20260806 12:28:14.714775 29836 leader_election.cc:302] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f37118653f144219c7b15a25b97d8b0; no voters:
I20260806 12:28:14.715171 29836 leader_election.cc:288] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:14.715271 29850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:14.715492 29850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [term 1 LEADER]: Becoming Leader. State: Replica: 2f37118653f144219c7b15a25b97d8b0, State: Running, Role: LEADER
I20260806 12:28:14.715888 29850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f37118653f144219c7b15a25b97d8b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } }
I20260806 12:28:14.717149 29836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:14.718155 29852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f37118653f144219c7b15a25b97d8b0. Latest consensus state: current_term: 1 leader_uuid: "2f37118653f144219c7b15a25b97d8b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f37118653f144219c7b15a25b97d8b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } } }
I20260806 12:28:14.718364 29852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:14.718744 29853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f37118653f144219c7b15a25b97d8b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f37118653f144219c7b15a25b97d8b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } } }
I20260806 12:28:14.718863 29853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:14.719506 29864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:14.723259 29864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:14.726732 29840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:14.715625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:33807 (local address 127.0.0.1:44625)
0806 12:28:14.715769 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:14.715774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:14.716195 (+ 421us) server_negotiation.cc:407] Connection header received
0806 12:28:14.716422 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:14.716425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:14.716485 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:14.716724 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:14.718013 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:14.718797 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:14.720211 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:14.720567 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:14.720642 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:14.722635 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:14.722639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:14.722642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:14.722677 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:14.724478 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:14.725190 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:14.725194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:14.725196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:14.725301 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:14.725694 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:14.725698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:14.725700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:14.725952 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:14.726118 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:14.726451 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 12:28:14.726567 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 12:28:14.730597 29864 catalog_manager.cc:1347] Generated new cluster ID: 204e567baef24c1bb0c75d89ee5f5248
I20260806 12:28:14.730684 29864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:14.745258 29864 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:14.746680 29864 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:14.758263 29864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f37118653f144219c7b15a25b97d8b0: Generated new TSK 0
I20260806 12:28:14.759374 29864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:14.768939 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 29722
W20260806 12:28:14.789490 29695 connection.cc:535] client connection to 127.0.0.1:44625 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44625): Connection reset by peer (error 104)
2026-08-06T12:28:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1654 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:14 dist-test-slave-lgr5 krb5kdc[29879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:14 dist-test-slave-lgr5 krb5kdc[29879](info): set up 2 sockets
Aug 06 12:28:14 dist-test-slave-lgr5 krb5kdc[29879](info): commencing operation
krb5kdc: starting...
W20260806 12:28:15.988848 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:16 dist-test-slave-lgr5 krb5kdc[29879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 12:28:16.141088 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:16.269855 29896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:16.270182 29896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:16.270227 29896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:16.274407 29896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:16.274505 29896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:16.274525 29896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:16.274541 29896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:16.274556 29896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:16.280557 29896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:16.281865 29896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:16.283015 29896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:16.293121 29903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:16.293216 29896 server_base.cc:1034] running on GCE node
W20260806 12:28:16.293124 29904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:16.293124 29907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:16.294181 29896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:16.295603 29896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:16.296823 29896 hybrid_clock.cc:648] HybridClock initialized: now 1786019296296803 us; error 36 us; skew 500 ppm
Aug 06 12:28:16 dist-test-slave-lgr5 krb5kdc[29879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:16.300107 29896 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 12:28:16.301149 29896 webserver.cc:460] Webserver started at http://127.0.0.1:32979/ using document root <none> and password file <none>
I20260806 12:28:16.301863 29896 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:16.301945 29896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:16.302136 29896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:16.304870 29896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "705d986277e24094b492b03a1741af1f"
format_stamp: "Formatted at 2026-08-06 12:28:16 on dist-test-slave-lgr5"
server_key: "c6773e2154248bf1d55dc029291769d5"
server_key_iv: "8af243562befe4708107c4211bae3223"
server_key_version: "encryptionkey@0"
I20260806 12:28:16.305658 29896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "705d986277e24094b492b03a1741af1f"
format_stamp: "Formatted at 2026-08-06 12:28:16 on dist-test-slave-lgr5"
server_key: "c6773e2154248bf1d55dc029291769d5"
server_key_iv: "8af243562befe4708107c4211bae3223"
server_key_version: "encryptionkey@0"
I20260806 12:28:16.311240 29896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 12:28:16.314415 29917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:16.315943 29896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 12:28:16.316179 29896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "705d986277e24094b492b03a1741af1f"
format_stamp: "Formatted at 2026-08-06 12:28:16 on dist-test-slave-lgr5"
server_key: "c6773e2154248bf1d55dc029291769d5"
server_key_iv: "8af243562befe4708107c4211bae3223"
server_key_version: "encryptionkey@0"
I20260806 12:28:16.316344 29896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:16.331053 29896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:16.334496 29896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:16.334698 29896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:16.342212 29896 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42127
I20260806 12:28:16.342253 30001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42127 every 8 connection(s)
I20260806 12:28:16.343330 29896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:16.346504 30002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:16.350463 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 29896
I20260806 12:28:16.350636 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:16.350961 29513 external_mini_cluster.cc:1428] Setting key ec5d140b7e0ea1dbff77ea03033d43ff
I20260806 12:28:16.353533 30002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f: Bootstrap starting.
I20260806 12:28:16.356302 30002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:16.357352 30002 log.cc:824] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:16.360008 30002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f: No bootstrap required, opened a new log
I20260806 12:28:16.361662 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:16.352615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:47869 (local address 127.0.0.1:42127)
0806 12:28:16.353107 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:16.353125 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:16.353166 (+ 41us) server_negotiation.cc:407] Connection header received
0806 12:28:16.353924 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:16.353947 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:16.354346 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:16.354668 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:16.355113 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:16.356240 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:16.357544 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:16.358215 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:16.358401 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:16.359465 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:16.359472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:16.359488 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:16.359954 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:16.360030 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:16.360059 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:16.360259 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:16.360538 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 12:28:16.360703 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":140,"threads_started":1}
I20260806 12:28:16.364049 30002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705d986277e24094b492b03a1741af1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42127 } }
I20260806 12:28:16.364342 30002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:16.364398 30002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705d986277e24094b492b03a1741af1f, State: Initialized, Role: FOLLOWER
I20260806 12:28:16.364880 30002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705d986277e24094b492b03a1741af1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42127 } }
I20260806 12:28:16.365000 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:16.365048 30002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:16.365114 30002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:16.365211 30002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:16.366539 30002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705d986277e24094b492b03a1741af1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42127 } }
I20260806 12:28:16.366953 30002 leader_election.cc:302] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 705d986277e24094b492b03a1741af1f; no voters:
I20260806 12:28:16.367456 30018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:16.367455 30002 leader_election.cc:288] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:16.367690 30018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [term 1 LEADER]: Becoming Leader. State: Replica: 705d986277e24094b492b03a1741af1f, State: Running, Role: LEADER
I20260806 12:28:16.368041 30018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705d986277e24094b492b03a1741af1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42127 } }
I20260806 12:28:16.368872 30002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:16.369841 30023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 705d986277e24094b492b03a1741af1f. Latest consensus state: current_term: 1 leader_uuid: "705d986277e24094b492b03a1741af1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705d986277e24094b492b03a1741af1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42127 } } }
I20260806 12:28:16.369977 30023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:16.370299 30022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "705d986277e24094b492b03a1741af1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705d986277e24094b492b03a1741af1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42127 } } }
I20260806 12:28:16.370396 30022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:16.370689 30037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:16.372681 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:16.367363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:38811 (local address 127.0.0.1:42127)
0806 12:28:16.367481 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:16.367486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:16.367712 (+ 226us) server_negotiation.cc:407] Connection header received
0806 12:28:16.367861 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:16.367864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:16.367929 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:16.368003 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:16.368451 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:16.369235 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:16.370381 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:16.370696 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:16.370776 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:16.372105 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:16.372109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:16.372112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:16.372127 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:16.372157 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:16.372171 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:16.372266 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:16.372506 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 12:28:16.372547 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 12:28:16.374368 30037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:16.380025 30037 catalog_manager.cc:1347] Generated new cluster ID: 9e163cb7092c4815bd20324ba7fc6f5b
I20260806 12:28:16.380126 30037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:16.394356 30037 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:16.395931 30037 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:16.412043 30037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 705d986277e24094b492b03a1741af1f: Generated new TSK 0
I20260806 12:28:16.412984 30037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:16.425551 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 29896
W20260806 12:28:16.441823 29876 connection.cc:535] client connection to 127.0.0.1:42127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42127): Connection reset by peer (error 104)
2026-08-06T12:28:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1652 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:16 dist-test-slave-lgr5 krb5kdc[30055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:16 dist-test-slave-lgr5 krb5kdc[30055](info): set up 2 sockets
Aug 06 12:28:16 dist-test-slave-lgr5 krb5kdc[30055](info): commencing operation
krb5kdc: starting...
W20260806 12:28:17.620783 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:17 dist-test-slave-lgr5 krb5kdc[30055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 12:28:17.767282 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:17.903827 30070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:17.904130 30070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:17.904172 30070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:17.908154 30070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:17.908218 30070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:17.908237 30070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:17.908252 30070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:17.908267 30070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:17.913663 30070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:17.914813 30070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:17.915889 30070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:17.925598 30079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:17.925628 30090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:17.925601 30087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:17.925887 30070 server_base.cc:1034] running on GCE node
I20260806 12:28:17.926609 30070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:17.927889 30070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:17.929112 30070 hybrid_clock.cc:648] HybridClock initialized: now 1786019297929094 us; error 38 us; skew 500 ppm
Aug 06 12:28:17 dist-test-slave-lgr5 krb5kdc[30055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:17.931872 30070 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 12:28:17.932860 30070 webserver.cc:460] Webserver started at http://127.0.0.1:41653/ using document root <none> and password file <none>
I20260806 12:28:17.933401 30070 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:17.933458 30070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:17.933624 30070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:17.935698 30070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "6412be20384841f494530537715864f3"
format_stamp: "Formatted at 2026-08-06 12:28:17 on dist-test-slave-lgr5"
server_key: "bed6d360da1427dc56721a793b9a9e94"
server_key_iv: "b7b3e99a05610c8273b50238d6e2f8f5"
server_key_version: "encryptionkey@0"
I20260806 12:28:17.936249 30070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "6412be20384841f494530537715864f3"
format_stamp: "Formatted at 2026-08-06 12:28:17 on dist-test-slave-lgr5"
server_key: "bed6d360da1427dc56721a793b9a9e94"
server_key_iv: "b7b3e99a05610c8273b50238d6e2f8f5"
server_key_version: "encryptionkey@0"
I20260806 12:28:17.939934 30070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 12:28:17.942385 30098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:17.943768 30070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 12:28:17.943897 30070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "6412be20384841f494530537715864f3"
format_stamp: "Formatted at 2026-08-06 12:28:17 on dist-test-slave-lgr5"
server_key: "bed6d360da1427dc56721a793b9a9e94"
server_key_iv: "b7b3e99a05610c8273b50238d6e2f8f5"
server_key_version: "encryptionkey@0"
I20260806 12:28:17.943997 30070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:17.948990 30070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:17.952041 30070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:17.952294 30070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:17.959733 30070 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43911
I20260806 12:28:17.959847 30178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43911 every 8 connection(s)
I20260806 12:28:17.960947 30070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:17.965472 30179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:17.969537 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 30070
I20260806 12:28:17.969679 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:17.970069 29513 external_mini_cluster.cc:1428] Setting key 94fcf94af03e0df67c58305311b0b4be
I20260806 12:28:17.973100 30179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3: Bootstrap starting.
I20260806 12:28:17.976527 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:17.971690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:60005 (local address 127.0.0.1:43911)
0806 12:28:17.972295 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:17.972313 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:17.972385 (+ 72us) server_negotiation.cc:407] Connection header received
0806 12:28:17.973255 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:17.973283 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:17.973687 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:17.974084 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:17.974905 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:17.974910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:17.974926 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:17.974938 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:17.975008 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:17.975050 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:17.975196 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:17.975268 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 12:28:17.975469 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":149,"threads_started":1}
I20260806 12:28:17.977169 30179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:17.978319 30179 log.cc:824] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:17.980932 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:17.981248 30179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3: No bootstrap required, opened a new log
I20260806 12:28:17.985610 30179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6412be20384841f494530537715864f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } }
I20260806 12:28:17.985899 30179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:17.985946 30179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6412be20384841f494530537715864f3, State: Initialized, Role: FOLLOWER
I20260806 12:28:17.986433 30179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6412be20384841f494530537715864f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } }
I20260806 12:28:17.986588 30179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:17.986660 30179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:17.986766 30179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:17.987847 30179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6412be20384841f494530537715864f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } }
I20260806 12:28:17.988289 30179 leader_election.cc:302] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6412be20384841f494530537715864f3; no voters:
I20260806 12:28:17.988703 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:17.983654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.158:55601 (local address 127.0.0.1:43911)
0806 12:28:17.983821 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:17.983826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:17.984124 (+ 298us) server_negotiation.cc:407] Connection header received
0806 12:28:17.984263 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:17.984266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:17.984316 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:17.984422 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:17.988153 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:17.988157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:17.988160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:17.988175 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:17.988213 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:17.988228 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:17.988337 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:17.988518 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 12:28:17.988569 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 12:28:17.988703 30179 leader_election.cc:288] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:17.988845 30195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:17.989135 30195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [term 1 LEADER]: Becoming Leader. State: Replica: 6412be20384841f494530537715864f3, State: Running, Role: LEADER
I20260806 12:28:17.989507 30195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6412be20384841f494530537715864f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } }
I20260806 12:28:17.990217 30179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:17.991375 30196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6412be20384841f494530537715864f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6412be20384841f494530537715864f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } } }
I20260806 12:28:17.991389 30198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6412be20384841f494530537715864f3. Latest consensus state: current_term: 1 leader_uuid: "6412be20384841f494530537715864f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6412be20384841f494530537715864f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } } }
I20260806 12:28:17.991537 30196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:17.991539 30198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:17.992156 30208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:17.995142 30208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:18.001379 30208 catalog_manager.cc:1347] Generated new cluster ID: 871e00a447cf4b579d142e2e01c04aaf
I20260806 12:28:18.001480 30208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:18.019335 30208 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:18.019763 30208 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:18.029605 30208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6412be20384841f494530537715864f3: Generated new TSK 0
I20260806 12:28:18.030616 30208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:18.044255 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 30070
2026-08-06T12:28:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1626 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:18 dist-test-slave-lgr5 krb5kdc[30228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:18 dist-test-slave-lgr5 krb5kdc[30228](info): set up 2 sockets
Aug 06 12:28:18 dist-test-slave-lgr5 krb5kdc[30228](info): commencing operation
krb5kdc: starting...
W20260806 12:28:19.250593 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:19 dist-test-slave-lgr5 krb5kdc[30228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:28:19.389600 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:44415
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:44415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:19.519820 30246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:19.520200 30246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:19.520252 30246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:19.524495 30246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:19.524559 30246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:19.524578 30246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:19.524593 30246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:19.524607 30246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:19.530351 30246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:44415
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:44415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:19.531584 30246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:19.532701 30246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:19.542358 30256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:19.542233 30262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:19.542536 30246 server_base.cc:1034] running on GCE node
W20260806 12:28:19.542428 30254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:19.543586 30246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:19.544987 30246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:19.546185 30246 hybrid_clock.cc:648] HybridClock initialized: now 1786019299546165 us; error 40 us; skew 500 ppm
Aug 06 12:28:19 dist-test-slave-lgr5 krb5kdc[30228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:19.549507 30246 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:28:19.550604 30246 webserver.cc:460] Webserver started at http://127.28.210.126:33403/ using document root <none> and password file <none>
I20260806 12:28:19.551298 30246 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:19.551373 30246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:19.551601 30246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:19.553995 30246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "ac3bf371354e4d46af8092f553e5af92"
format_stamp: "Formatted at 2026-08-06 12:28:19 on dist-test-slave-lgr5"
server_key: "a99af12824df51d67abab26ee0c97e80"
server_key_iv: "ea294156113aa4041ea29c8494da60b2"
server_key_version: "encryptionkey@0"
I20260806 12:28:19.554639 30246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "ac3bf371354e4d46af8092f553e5af92"
format_stamp: "Formatted at 2026-08-06 12:28:19 on dist-test-slave-lgr5"
server_key: "a99af12824df51d67abab26ee0c97e80"
server_key_iv: "ea294156113aa4041ea29c8494da60b2"
server_key_version: "encryptionkey@0"
I20260806 12:28:19.559127 30246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 12:28:19.562265 30274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:19.564060 30246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 12:28:19.564265 30246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "ac3bf371354e4d46af8092f553e5af92"
format_stamp: "Formatted at 2026-08-06 12:28:19 on dist-test-slave-lgr5"
server_key: "a99af12824df51d67abab26ee0c97e80"
server_key_iv: "ea294156113aa4041ea29c8494da60b2"
server_key_version: "encryptionkey@0"
I20260806 12:28:19.564388 30246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:19.586519 30246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:19.590788 30246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:19.591032 30246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:19.599236 30246 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:44415
I20260806 12:28:19.599258 30353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:44415 every 8 connection(s)
I20260806 12:28:19.600335 30246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:19.603356 30354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:19.608618 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 30246
I20260806 12:28:19.608788 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:19.609097 29513 external_mini_cluster.cc:1428] Setting key 83b0db020ef57bfc50909844cae354aa
I20260806 12:28:19.610929 30354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92: Bootstrap starting.
I20260806 12:28:19.613668 30354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:19.614454 30354 log.cc:824] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:19.616815 30354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92: No bootstrap required, opened a new log
Aug 06 12:28:19 dist-test-slave-lgr5 krb5kdc[30228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:28:19.621093 30354 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3bf371354e4d46af8092f553e5af92" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44415 } }
I20260806 12:28:19.621430 30354 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:19.621515 30354 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3bf371354e4d46af8092f553e5af92, State: Initialized, Role: FOLLOWER
I20260806 12:28:19.622052 30354 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3bf371354e4d46af8092f553e5af92" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44415 } }
I20260806 12:28:19.622224 30354 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:19.622334 30354 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:19.622462 30354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:19.623927 30354 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3bf371354e4d46af8092f553e5af92" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44415 } }
I20260806 12:28:19.624390 30354 leader_election.cc:302] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3bf371354e4d46af8092f553e5af92; no voters:
I20260806 12:28:19.624832 30354 leader_election.cc:288] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:19.624934 30362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:19.625131 30362 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [term 1 LEADER]: Becoming Leader. State: Replica: ac3bf371354e4d46af8092f553e5af92, State: Running, Role: LEADER
I20260806 12:28:19.625504 30362 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3bf371354e4d46af8092f553e5af92" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44415 } }
I20260806 12:28:19.625759 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:19.610667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49487 (local address 127.28.210.126:44415)
0806 12:28:19.611131 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:19.611148 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:19.611187 (+ 39us) server_negotiation.cc:407] Connection header received
0806 12:28:19.611894 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:19.611912 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:19.612360 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:19.612654 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:19.613602 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:19.614780 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:19.615856 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:19.616443 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:19.616794 (+ 351us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:19.619809 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:19.619817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:19.619839 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:19.619882 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:19.621743 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:19.622438 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:19.622447 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:19.622455 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:19.622643 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:19.623026 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:19.623030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:19.623032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:19.623701 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:19.624006 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:19.624378 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 12:28:19.624682 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":136,"threads_started":1}
I20260806 12:28:19.626494 30354 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:19.628101 30364 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac3bf371354e4d46af8092f553e5af92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3bf371354e4d46af8092f553e5af92" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44415 } } }
I20260806 12:28:19.628260 30366 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac3bf371354e4d46af8092f553e5af92. Latest consensus state: current_term: 1 leader_uuid: "ac3bf371354e4d46af8092f553e5af92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3bf371354e4d46af8092f553e5af92" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44415 } } }
I20260806 12:28:19.628340 30366 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:19.628322 30364 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:19.629237 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 12:28:19.632400 30385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:28:19.632506 30385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:28:19.632570 30380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:19.633473 30380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:19.640612 30380 catalog_manager.cc:1347] Generated new cluster ID: 55749d7333d24a6787dc28c3965058fc
I20260806 12:28:19.640702 30380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:19.641414 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:19.631664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48125 (local address 127.28.210.126:44415)
0806 12:28:19.631784 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:19.631788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:19.632427 (+ 639us) server_negotiation.cc:407] Connection header received
0806 12:28:19.632581 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:19.632584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:19.632636 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:19.632689 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:19.633492 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:19.634058 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:19.634875 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:19.635112 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:19.635200 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:19.636940 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:19.636944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:19.636947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:19.636975 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:19.638974 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:19.639551 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:19.639555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:19.639556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:19.639647 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:19.640005 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:19.640009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:19.640011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:19.640305 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:19.640541 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:19.641178 (+ 637us) server_negotiation.cc:299] Negotiation successful
0806 12:28:19.641287 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 12:28:19.667272 30380 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:19.668500 30380 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:19.673045 30380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac3bf371354e4d46af8092f553e5af92: Generated new TSK 0
I20260806 12:28:19.673851 30380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:19.686295 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 30246
W20260806 12:28:19.706401 30223 connection.cc:535] client connection to 127.28.210.126:44415 recv error: Network error: failed to read from TLS socket (remote: 127.28.210.126:44415): Connection reset by peer (error 104)
2026-08-06T12:28:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1638 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:19 dist-test-slave-lgr5 krb5kdc[30401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:19 dist-test-slave-lgr5 krb5kdc[30401](info): set up 2 sockets
Aug 06 12:28:19 dist-test-slave-lgr5 krb5kdc[30401](info): commencing operation
krb5kdc: starting...
W20260806 12:28:20.915031 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:21 dist-test-slave-lgr5 krb5kdc[30401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:28:21.065294 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:35153
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:44511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:35153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:21.187722 30417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:21.188033 30417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:21.188124 30417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:21.192194 30417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:21.192269 30417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:21.192296 30417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:21.192317 30417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:21.192338 30417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:21.197892 30417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:44511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:35153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:35153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:21.199057 30417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:21.200132 30417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:21.209590 30423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:21.209658 30429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:21.209678 30417 server_base.cc:1034] running on GCE node
W20260806 12:28:21.209561 30424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:21.210613 30417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:21.211879 30417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:21.213061 30417 hybrid_clock.cc:648] HybridClock initialized: now 1786019301213061 us; error 45 us; skew 500 ppm
Aug 06 12:28:21 dist-test-slave-lgr5 krb5kdc[30401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019301, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:21.216010 30417 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:28:21.217113 30417 webserver.cc:460] Webserver started at http://127.28.210.126:45847/ using document root <none> and password file <none>
I20260806 12:28:21.217784 30417 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:21.217878 30417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:21.218169 30417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:21.220829 30417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "2bf390dd4d514da7b1c0aa9bfbb43248"
format_stamp: "Formatted at 2026-08-06 12:28:21 on dist-test-slave-lgr5"
server_key: "4193bec449f62257c202ab9d2262e6a1"
server_key_iv: "8e6c23ccc9963f753bebad6cd41273b9"
server_key_version: "encryptionkey@0"
I20260806 12:28:21.221501 30417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "2bf390dd4d514da7b1c0aa9bfbb43248"
format_stamp: "Formatted at 2026-08-06 12:28:21 on dist-test-slave-lgr5"
server_key: "4193bec449f62257c202ab9d2262e6a1"
server_key_iv: "8e6c23ccc9963f753bebad6cd41273b9"
server_key_version: "encryptionkey@0"
I20260806 12:28:21.226076 30417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 12:28:21.229090 30445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:21.230618 30417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:28:21.230806 30417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "2bf390dd4d514da7b1c0aa9bfbb43248"
format_stamp: "Formatted at 2026-08-06 12:28:21 on dist-test-slave-lgr5"
server_key: "4193bec449f62257c202ab9d2262e6a1"
server_key_iv: "8e6c23ccc9963f753bebad6cd41273b9"
server_key_version: "encryptionkey@0"
I20260806 12:28:21.230947 30417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:21.254138 30417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:21.257972 30417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:21.258189 30417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:21.265776 30417 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:35153
I20260806 12:28:21.265806 30525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:35153 every 8 connection(s)
I20260806 12:28:21.266866 30417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:21.269975 30526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:21.274035 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 30417
I20260806 12:28:21.274200 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:21.274530 29513 external_mini_cluster.cc:1428] Setting key 6bb994ee63dc087de82881b70848cc8b
I20260806 12:28:21.276749 30526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248: Bootstrap starting.
I20260806 12:28:21.279561 30526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:21.280443 30526 log.cc:824] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:21.282974 30526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248: No bootstrap required, opened a new log
I20260806 12:28:21.285580 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:21.276310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52227 (local address 127.28.210.126:35153)
0806 12:28:21.276896 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:21.276936 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:21.276985 (+ 49us) server_negotiation.cc:407] Connection header received
0806 12:28:21.277713 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:21.277733 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:21.278143 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:21.278526 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:21.279007 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:21.280132 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:21.281221 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:21.281880 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:21.282088 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:21.283218 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:21.283224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:21.283240 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:21.283706 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:21.283778 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:21.283809 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:21.283939 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:21.284304 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 12:28:21.284584 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":184,"threads_started":1}
I20260806 12:28:21.287472 30526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf390dd4d514da7b1c0aa9bfbb43248" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35153 } }
I20260806 12:28:21.287781 30526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:21.287858 30526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf390dd4d514da7b1c0aa9bfbb43248, State: Initialized, Role: FOLLOWER
I20260806 12:28:21.288419 30526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf390dd4d514da7b1c0aa9bfbb43248" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35153 } }
I20260806 12:28:21.288609 30526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:21.288686 30526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:21.288797 30526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:21.289817 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:21.290189 30526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf390dd4d514da7b1c0aa9bfbb43248" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35153 } }
I20260806 12:28:21.290678 30526 leader_election.cc:302] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bf390dd4d514da7b1c0aa9bfbb43248; no voters:
I20260806 12:28:21.291174 30526 leader_election.cc:288] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:21.291355 30533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:21.291790 30533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [term 1 LEADER]: Becoming Leader. State: Replica: 2bf390dd4d514da7b1c0aa9bfbb43248, State: Running, Role: LEADER
I20260806 12:28:21.292272 30533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf390dd4d514da7b1c0aa9bfbb43248" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35153 } }
I20260806 12:28:21.292853 30526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:21.294353 30540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bf390dd4d514da7b1c0aa9bfbb43248. Latest consensus state: current_term: 1 leader_uuid: "2bf390dd4d514da7b1c0aa9bfbb43248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf390dd4d514da7b1c0aa9bfbb43248" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35153 } } }
I20260806 12:28:21.294672 30540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:21.295085 30536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bf390dd4d514da7b1c0aa9bfbb43248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf390dd4d514da7b1c0aa9bfbb43248" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 35153 } } }
I20260806 12:28:21.295181 30536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:21.298578 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:21.292610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32991 (local address 127.28.210.126:35153)
0806 12:28:21.292749 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:21.292754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:21.293244 (+ 490us) server_negotiation.cc:407] Connection header received
0806 12:28:21.293318 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:21.293321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:21.293378 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:21.293472 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:21.293829 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:21.294701 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:21.296247 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:21.296570 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:21.296658 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:21.298003 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:21.298007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:21.298010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:21.298029 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:21.298066 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:21.298082 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:21.298183 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:21.298362 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 12:28:21.298409 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 12:28:21.299310 30563 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:28:21.299391 30563 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:28:21.299494 30558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:21.300439 30558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:21.307438 30558 catalog_manager.cc:1347] Generated new cluster ID: 54882f8482b54b0b906d6bef48bcb2f7
I20260806 12:28:21.307547 30558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:21.320758 30558 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:21.322031 30558 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:21.334537 30558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bf390dd4d514da7b1c0aa9bfbb43248: Generated new TSK 0
I20260806 12:28:21.335783 30558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:21.350507 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 30417
W20260806 12:28:21.362445 30399 connection.cc:535] client connection to 127.28.210.126:35153 recv error: Network error: failed to read from TLS socket (remote: 127.28.210.126:35153): Connection reset by peer (error 104)
2026-08-06T12:28:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1655 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:21 dist-test-slave-lgr5 krb5kdc[30571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:21 dist-test-slave-lgr5 krb5kdc[30571](info): set up 2 sockets
Aug 06 12:28:21 dist-test-slave-lgr5 krb5kdc[30571](info): commencing operation
krb5kdc: starting...
W20260806 12:28:22.568176 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:22 dist-test-slave-lgr5 krb5kdc[30571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:28:22.754457 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:38659
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:38659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:22.990356 30588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:22.991070 30588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:22.991179 30588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:22.998926 30588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:22.999101 30588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:22.999171 30588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:22.999226 30588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:22.999279 30588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:23.010910 30588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:38659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:38659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:23.013208 30588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:23.014736 30588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:23.032752 30597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:23.032794 30596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:23.032960 30604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:23.033236 30588 server_base.cc:1034] running on GCE node
I20260806 12:28:23.034771 30588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:23.037256 30588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:23.040838 30588 hybrid_clock.cc:648] HybridClock initialized: now 1786019303040701 us; error 137 us; skew 500 ppm
Aug 06 12:28:23 dist-test-slave-lgr5 krb5kdc[30571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019303, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:23.050789 30588 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:28:23.053093 30588 webserver.cc:460] Webserver started at http://127.28.210.126:43825/ using document root <none> and password file <none>
I20260806 12:28:23.054311 30588 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:23.054491 30588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:23.054965 30588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:23.060223 30588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "b5f2907223c541748c720b7a6d772794"
format_stamp: "Formatted at 2026-08-06 12:28:23 on dist-test-slave-lgr5"
server_key: "cc5076ab3d16502b7732694f8b2852e3"
server_key_iv: "583f21d02af94d3be6de445adff7afdd"
server_key_version: "encryptionkey@0"
I20260806 12:28:23.061669 30588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "b5f2907223c541748c720b7a6d772794"
format_stamp: "Formatted at 2026-08-06 12:28:23 on dist-test-slave-lgr5"
server_key: "cc5076ab3d16502b7732694f8b2852e3"
server_key_iv: "583f21d02af94d3be6de445adff7afdd"
server_key_version: "encryptionkey@0"
I20260806 12:28:23.070634 30588 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260806 12:28:23.077468 30613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:23.080483 30588 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260806 12:28:23.080727 30588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "b5f2907223c541748c720b7a6d772794"
format_stamp: "Formatted at 2026-08-06 12:28:23 on dist-test-slave-lgr5"
server_key: "cc5076ab3d16502b7732694f8b2852e3"
server_key_iv: "583f21d02af94d3be6de445adff7afdd"
server_key_version: "encryptionkey@0"
I20260806 12:28:23.080968 30588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:23.092700 30588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:23.098165 30588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:23.098479 30588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:23.112829 30588 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:38659
I20260806 12:28:23.112880 30700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:38659 every 8 connection(s)
I20260806 12:28:23.114908 30588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:23.120811 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 30588
I20260806 12:28:23.121186 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:23.121914 29513 external_mini_cluster.cc:1428] Setting key e67a5c81173c7a015d184365a10278c9
I20260806 12:28:23.122794 30701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:23.133529 30705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:23.125024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56227 (local address 127.28.210.126:38659)
0806 12:28:23.126123 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:23.126161 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:23.126244 (+ 83us) server_negotiation.cc:407] Connection header received
0806 12:28:23.127557 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:23.127599 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:23.128273 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:23.128906 (+ 633us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:23.130537 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:23.130545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:23.130582 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:23.130600 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:23.130753 (+ 153us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:23.130809 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:23.131139 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:23.131316 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 12:28:23.131642 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":312,"threads_started":1}
I20260806 12:28:23.136603 30701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794: Bootstrap starting.
I20260806 12:28:23.140256 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:23.143828 30701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:23.146471 30701 log.cc:824] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:23.150174 30705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:23.146317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49765 (local address 127.28.210.126:38659)
0806 12:28:23.146640 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:23.146647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:23.147070 (+ 423us) server_negotiation.cc:407] Connection header received
0806 12:28:23.147254 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:23.147259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:23.147417 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:23.147781 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:23.149362 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:23.149380 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:23.149384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:23.149424 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:23.149504 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:23.149527 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:23.149767 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:23.149912 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 12:28:23.150010 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260806 12:28:23.152499 30701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794: No bootstrap required, opened a new log
I20260806 12:28:23.164443 30701 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f2907223c541748c720b7a6d772794" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 38659 } }
I20260806 12:28:23.165206 30701 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:23.165292 30701 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f2907223c541748c720b7a6d772794, State: Initialized, Role: FOLLOWER
I20260806 12:28:23.165975 30701 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f2907223c541748c720b7a6d772794" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 38659 } }
I20260806 12:28:23.166210 30701 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:23.166296 30701 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:23.166469 30701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:23.168767 30701 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f2907223c541748c720b7a6d772794" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 38659 } }
I20260806 12:28:23.169377 30701 leader_election.cc:302] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5f2907223c541748c720b7a6d772794; no voters:
I20260806 12:28:23.169921 30701 leader_election.cc:288] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:23.170123 30718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:23.170558 30718 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [term 1 LEADER]: Becoming Leader. State: Replica: b5f2907223c541748c720b7a6d772794, State: Running, Role: LEADER
I20260806 12:28:23.171196 30718 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f2907223c541748c720b7a6d772794" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 38659 } }
I20260806 12:28:23.172457 30701 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:23.174916 30721 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5f2907223c541748c720b7a6d772794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f2907223c541748c720b7a6d772794" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 38659 } } }
I20260806 12:28:23.175231 30721 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:23.176999 30734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:23.177282 30723 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5f2907223c541748c720b7a6d772794. Latest consensus state: current_term: 1 leader_uuid: "b5f2907223c541748c720b7a6d772794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f2907223c541748c720b7a6d772794" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 38659 } } }
I20260806 12:28:23.177911 30723 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:23.184693 30734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:23.198904 30734 catalog_manager.cc:1347] Generated new cluster ID: 72a2e05feeee463289a7cf8faed31076
I20260806 12:28:23.199106 30734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:23.227178 30734 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:23.228370 30734 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:23.249257 30734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5f2907223c541748c720b7a6d772794: Generated new TSK 0
I20260806 12:28:23.250917 30734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:23.263083 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 30588
2026-08-06T12:28:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:23 dist-test-slave-lgr5 krb5kdc[30751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:23 dist-test-slave-lgr5 krb5kdc[30751](info): set up 2 sockets
Aug 06 12:28:23 dist-test-slave-lgr5 krb5kdc[30751](info): commencing operation
krb5kdc: starting...
W20260806 12:28:24.896060 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.585s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:25 dist-test-slave-lgr5 krb5kdc[30751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:28:25.063549 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:40819
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:45653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:40819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:25.196493 30768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:25.196861 30768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:25.196934 30768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:25.201257 30768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:25.201387 30768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:25.201419 30768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:25.201449 30768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:25.201484 30768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:25.207261 30768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:45653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:40819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:40819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:25.208585 30768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:25.209755 30768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:25.219578 30779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:25.219578 30774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:25.219537 30776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:25.219862 30768 server_base.cc:1034] running on GCE node
I20260806 12:28:25.220590 30768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:25.221863 30768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:25.223057 30768 hybrid_clock.cc:648] HybridClock initialized: now 1786019305223039 us; error 32 us; skew 500 ppm
Aug 06 12:28:25 dist-test-slave-lgr5 krb5kdc[30751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019305, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:25.226136 30768 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:28:25.227226 30768 webserver.cc:460] Webserver started at http://127.28.210.126:43821/ using document root <none> and password file <none>
I20260806 12:28:25.227837 30768 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:25.227905 30768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:25.228176 30768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:25.230502 30768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "f69103c4e9704c759e62ee3dcce113e3"
format_stamp: "Formatted at 2026-08-06 12:28:25 on dist-test-slave-lgr5"
server_key: "4b0250f4ec2a6ebe6ea8c83a4c824e00"
server_key_iv: "112781a78fd74637a24f530244940561"
server_key_version: "encryptionkey@0"
I20260806 12:28:25.231145 30768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "f69103c4e9704c759e62ee3dcce113e3"
format_stamp: "Formatted at 2026-08-06 12:28:25 on dist-test-slave-lgr5"
server_key: "4b0250f4ec2a6ebe6ea8c83a4c824e00"
server_key_iv: "112781a78fd74637a24f530244940561"
server_key_version: "encryptionkey@0"
I20260806 12:28:25.235689 30768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 12:28:25.238566 30787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:25.240370 30768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:28:25.240595 30768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "f69103c4e9704c759e62ee3dcce113e3"
format_stamp: "Formatted at 2026-08-06 12:28:25 on dist-test-slave-lgr5"
server_key: "4b0250f4ec2a6ebe6ea8c83a4c824e00"
server_key_iv: "112781a78fd74637a24f530244940561"
server_key_version: "encryptionkey@0"
I20260806 12:28:25.240752 30768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:25.261417 30768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:25.266173 30768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:25.266459 30768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:25.276151 30768 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:40819
I20260806 12:28:25.276223 30865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:40819 every 8 connection(s)
I20260806 12:28:25.277472 30768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:25.281570 30867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:25.283227 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 30768
I20260806 12:28:25.283393 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:25.283715 29513 external_mini_cluster.cc:1428] Setting key 61287adec60044944482e21066a8642a
I20260806 12:28:25.289482 30867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3: Bootstrap starting.
I20260806 12:28:25.293030 30867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:25.294183 30867 log.cc:824] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3: Log is configured to *not* fsync() on all Append() calls
Aug 06 12:28:25 dist-test-slave-lgr5 krb5kdc[30751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:28:25.297001 30867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3: No bootstrap required, opened a new log
I20260806 12:28:25.300961 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:25.285476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35683 (local address 127.28.210.126:40819)
0806 12:28:25.285980 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:25.285998 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:25.286041 (+ 43us) server_negotiation.cc:407] Connection header received
0806 12:28:25.286764 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:25.286785 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:25.287198 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:25.287535 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:25.288709 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:25.289898 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:25.291181 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:25.291774 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:25.291946 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:25.295282 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:25.295289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:25.295308 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:25.295345 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:25.297479 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:25.298063 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:25.298070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:25.298076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:25.298228 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:25.298552 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:25.298556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:25.298559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:25.299053 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:25.299298 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:25.299630 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 12:28:25.300005 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":154,"threads_started":1}
I20260806 12:28:25.301301 30867 raft_consensus.cc:348] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69103c4e9704c759e62ee3dcce113e3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40819 } }
I20260806 12:28:25.301604 30867 raft_consensus.cc:374] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:25.301666 30867 raft_consensus.cc:729] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69103c4e9704c759e62ee3dcce113e3, State: Initialized, Role: FOLLOWER
I20260806 12:28:25.302199 30867 consensus_queue.cc:260] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69103c4e9704c759e62ee3dcce113e3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40819 } }
I20260806 12:28:25.302354 30867 raft_consensus.cc:388] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:25.302431 30867 raft_consensus.cc:482] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:25.302544 30867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:25.303776 30867 raft_consensus.cc:504] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69103c4e9704c759e62ee3dcce113e3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40819 } }
I20260806 12:28:25.304234 30867 leader_election.cc:302] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f69103c4e9704c759e62ee3dcce113e3; no voters:
I20260806 12:28:25.304726 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:25.304751 30867 leader_election.cc:288] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:25.304883 30876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:25.305109 30876 raft_consensus.cc:686] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [term 1 LEADER]: Becoming Leader. State: Replica: f69103c4e9704c759e62ee3dcce113e3, State: Running, Role: LEADER
I20260806 12:28:25.305634 30876 consensus_queue.cc:237] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69103c4e9704c759e62ee3dcce113e3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40819 } }
I20260806 12:28:25.306444 30867 sys_catalog.cc:564] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:25.307682 30878 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f69103c4e9704c759e62ee3dcce113e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69103c4e9704c759e62ee3dcce113e3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40819 } } }
I20260806 12:28:25.307873 30878 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:25.308204 30878 sys_catalog.cc:455] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f69103c4e9704c759e62ee3dcce113e3. Latest consensus state: current_term: 1 leader_uuid: "f69103c4e9704c759e62ee3dcce113e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f69103c4e9704c759e62ee3dcce113e3" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40819 } } }
I20260806 12:28:25.308316 30878 sys_catalog.cc:458] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:25.308362 30896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:25.312335 30896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:25.319090 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:25.307579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55915 (local address 127.28.210.126:40819)
0806 12:28:25.307725 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:25.307730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:25.309189 (+ 1459us) server_negotiation.cc:407] Connection header received
0806 12:28:25.309344 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:25.309348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:25.309413 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:25.309520 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:25.310992 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:25.311549 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:25.312974 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:25.313291 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:25.313363 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:25.315092 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:25.315097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:25.315100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:25.315139 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:25.317054 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:25.317700 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:25.317703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:25.317705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:25.317801 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:25.318109 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:25.318113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:25.318115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:25.318396 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:25.318532 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:25.318787 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 12:28:25.318921 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:28:25.320570 30896 catalog_manager.cc:1347] Generated new cluster ID: eb8db66dc16d442eac50bd44c944398d
I20260806 12:28:25.320653 30896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:25.366436 30896 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:25.367580 30896 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:25.377017 30896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f69103c4e9704c759e62ee3dcce113e3: Generated new TSK 0
I20260806 12:28:25.378037 30896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:25.390868 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 30768
2026-08-06T12:28:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2121 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:25 dist-test-slave-lgr5 krb5kdc[30913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:25 dist-test-slave-lgr5 krb5kdc[30913](info): set up 2 sockets
Aug 06 12:28:25 dist-test-slave-lgr5 krb5kdc[30913](info): commencing operation
krb5kdc: starting...
W20260806 12:28:26.610504 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:26 dist-test-slave-lgr5 krb5kdc[30913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:28:26.760659 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:44153
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:37389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:44153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:26.892248 30932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:26.892570 30932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:26.892614 30932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:26.896976 30932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:26.897058 30932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:26.897076 30932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:26.897092 30932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:26.897106 30932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:26.902750 30932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:37389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:44153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:44153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:26.903903 30932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:26.905025 30932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:26.915081 30940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:26.915035 30943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:26.915057 30932 server_base.cc:1034] running on GCE node
W20260806 12:28:26.915061 30941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:26.916242 30932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:26.917668 30932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:26.918882 30932 hybrid_clock.cc:648] HybridClock initialized: now 1786019306918821 us; error 83 us; skew 500 ppm
Aug 06 12:28:26 dist-test-slave-lgr5 krb5kdc[30913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019306, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:26.922353 30932 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:28:26.923501 30932 webserver.cc:460] Webserver started at http://127.28.210.126:33543/ using document root <none> and password file <none>
I20260806 12:28:26.924220 30932 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:26.924311 30932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:26.924557 30932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:26.927038 30932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "67ab5b7f1eaf47158bf79e4a98c85334"
format_stamp: "Formatted at 2026-08-06 12:28:26 on dist-test-slave-lgr5"
server_key: "24816d45f8f563b49ac1afca506d5cbb"
server_key_iv: "9ba1527a82a41c5b184c66aa3d570a9d"
server_key_version: "encryptionkey@0"
I20260806 12:28:26.927703 30932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "67ab5b7f1eaf47158bf79e4a98c85334"
format_stamp: "Formatted at 2026-08-06 12:28:26 on dist-test-slave-lgr5"
server_key: "24816d45f8f563b49ac1afca506d5cbb"
server_key_iv: "9ba1527a82a41c5b184c66aa3d570a9d"
server_key_version: "encryptionkey@0"
I20260806 12:28:26.932432 30932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 12:28:26.935489 30950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:26.937129 30932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:28:26.937297 30932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "67ab5b7f1eaf47158bf79e4a98c85334"
format_stamp: "Formatted at 2026-08-06 12:28:26 on dist-test-slave-lgr5"
server_key: "24816d45f8f563b49ac1afca506d5cbb"
server_key_iv: "9ba1527a82a41c5b184c66aa3d570a9d"
server_key_version: "encryptionkey@0"
I20260806 12:28:26.937439 30932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:26.959699 30932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:26.963503 30932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:26.963727 30932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:26.971920 30932 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:44153
I20260806 12:28:26.971940 31024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:44153 every 8 connection(s)
I20260806 12:28:26.973286 30932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:26.977509 31027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:26.981477 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 30932
I20260806 12:28:26.981662 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:26.982010 29513 external_mini_cluster.cc:1428] Setting key 0eab476fd2df499eb0eb85e07a477691
I20260806 12:28:26.985954 31027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334: Bootstrap starting.
I20260806 12:28:26.988773 31027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:26.989707 31027 log.cc:824] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:26.992719 31027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334: No bootstrap required, opened a new log
I20260806 12:28:26.995036 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:26.984462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52147 (local address 127.28.210.126:44153)
0806 12:28:26.985117 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:26.985135 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:26.985383 (+ 248us) server_negotiation.cc:407] Connection header received
0806 12:28:26.986768 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:26.986805 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:26.987272 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:26.987688 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:26.988276 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:26.989483 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:26.990659 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:26.991465 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:26.991681 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:26.992647 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:26.992655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:26.992675 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:26.993289 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:26.993370 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:26.993408 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:26.993538 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:26.993701 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 12:28:26.993950 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":195,"threads_started":1}
I20260806 12:28:26.997891 31027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ab5b7f1eaf47158bf79e4a98c85334" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44153 } }
I20260806 12:28:26.998212 31027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:26.998265 31027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ab5b7f1eaf47158bf79e4a98c85334, State: Initialized, Role: FOLLOWER
I20260806 12:28:26.998519 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:26.998816 31027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ab5b7f1eaf47158bf79e4a98c85334" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44153 } }
I20260806 12:28:26.998976 31027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:26.999050 31027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:26.999156 31027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:27.000581 31027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ab5b7f1eaf47158bf79e4a98c85334" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44153 } }
I20260806 12:28:27.001204 31027 leader_election.cc:302] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67ab5b7f1eaf47158bf79e4a98c85334; no voters:
I20260806 12:28:27.001713 31027 leader_election.cc:288] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:27.001880 31050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:27.002146 31050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [term 1 LEADER]: Becoming Leader. State: Replica: 67ab5b7f1eaf47158bf79e4a98c85334, State: Running, Role: LEADER
I20260806 12:28:27.002635 31050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ab5b7f1eaf47158bf79e4a98c85334" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44153 } }
I20260806 12:28:27.003490 31027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:27.004647 31051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67ab5b7f1eaf47158bf79e4a98c85334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ab5b7f1eaf47158bf79e4a98c85334" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44153 } } }
I20260806 12:28:27.004840 31051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:27.005244 31052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67ab5b7f1eaf47158bf79e4a98c85334. Latest consensus state: current_term: 1 leader_uuid: "67ab5b7f1eaf47158bf79e4a98c85334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ab5b7f1eaf47158bf79e4a98c85334" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 44153 } } }
I20260806 12:28:27.005353 31052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:27.006829 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:27.001126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45661 (local address 127.28.210.126:44153)
0806 12:28:27.001293 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:27.001299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:27.001502 (+ 203us) server_negotiation.cc:407] Connection header received
0806 12:28:27.001647 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:27.001651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:27.001722 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:27.001834 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:27.002473 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:27.003267 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:27.004503 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:27.004856 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:27.004953 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:27.006290 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:27.006294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:27.006297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:27.006314 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:27.006349 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:27.006364 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:27.006433 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:27.006593 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 12:28:27.006666 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 12:28:27.010022 31073 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:28:27.010143 31073 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:28:27.010257 31065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:27.011055 31065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:27.017856 31065 catalog_manager.cc:1347] Generated new cluster ID: 83a23e39521944d5863d9f3c00b64126
I20260806 12:28:27.017957 31065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:27.033411 31065 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:27.034471 31065 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:27.046221 31065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67ab5b7f1eaf47158bf79e4a98c85334: Generated new TSK 0
I20260806 12:28:27.047278 31065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:27.060763 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 30932
2026-08-06T12:28:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1660 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:27 dist-test-slave-lgr5 krb5kdc[31082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:27 dist-test-slave-lgr5 krb5kdc[31082](info): set up 2 sockets
Aug 06 12:28:27 dist-test-slave-lgr5 krb5kdc[31082](info): commencing operation
krb5kdc: starting...
W20260806 12:28:28.267040 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:28 dist-test-slave-lgr5 krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:28:28.413762 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:38971
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:38971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:28.548064 31100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:28.548457 31100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:28.548537 31100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:28.553179 31100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:28.553300 31100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:28.553330 31100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:28.553355 31100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:28.553387 31100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:28.560534 31100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:38971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:38971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:28.562100 31100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:28.563351 31100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:28.573208 31113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:28.573154 31111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:28.573172 31110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:28.573161 31100 server_base.cc:1034] running on GCE node
I20260806 12:28:28.574112 31100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:28.575443 31100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:28.576670 31100 hybrid_clock.cc:648] HybridClock initialized: now 1786019308576646 us; error 42 us; skew 500 ppm
Aug 06 12:28:28 dist-test-slave-lgr5 krb5kdc[31082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:28.583915 31100 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:28:28.585187 31100 webserver.cc:460] Webserver started at http://127.28.210.126:36891/ using document root <none> and password file <none>
I20260806 12:28:28.585814 31100 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:28.585889 31100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:28.586112 31100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:28.588541 31100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "b63ce4f9c468421f8f97c6492622e124"
format_stamp: "Formatted at 2026-08-06 12:28:28 on dist-test-slave-lgr5"
server_key: "47b2a85b229d2e06c2c85eb58300f071"
server_key_iv: "49836a747618735867063744bf271ade"
server_key_version: "encryptionkey@0"
I20260806 12:28:28.589183 31100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "b63ce4f9c468421f8f97c6492622e124"
format_stamp: "Formatted at 2026-08-06 12:28:28 on dist-test-slave-lgr5"
server_key: "47b2a85b229d2e06c2c85eb58300f071"
server_key_iv: "49836a747618735867063744bf271ade"
server_key_version: "encryptionkey@0"
I20260806 12:28:28.593863 31100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 12:28:28.596928 31123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:28.598489 31100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:28:28.598658 31100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "b63ce4f9c468421f8f97c6492622e124"
format_stamp: "Formatted at 2026-08-06 12:28:28 on dist-test-slave-lgr5"
server_key: "47b2a85b229d2e06c2c85eb58300f071"
server_key_iv: "49836a747618735867063744bf271ade"
server_key_version: "encryptionkey@0"
I20260806 12:28:28.598790 31100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:28.603873 31100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:28.607312 31100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:28.607503 31100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:28.616935 31100 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:38971
I20260806 12:28:28.617045 31205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:38971 every 8 connection(s)
I20260806 12:28:28.618266 31100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:28.621996 31206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:28.623150 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 31100
I20260806 12:28:28.623323 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:28.623628 29513 external_mini_cluster.cc:1428] Setting key 6d98827108b7042ce8e2749fa92ada5b
I20260806 12:28:28.630389 31206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124: Bootstrap starting.
I20260806 12:28:28.631091 31210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:28.625377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41085 (local address 127.28.210.126:38971)
0806 12:28:28.626001 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:28.626019 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:28.626067 (+ 48us) server_negotiation.cc:407] Connection header received
0806 12:28:28.626848 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:28.626870 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:28.627212 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:28.627615 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:28.628905 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:28.628913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:28.628932 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:28.628966 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:28.629054 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:28.629087 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:28.629258 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:28.629515 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 12:28:28.629946 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":187,"threads_started":1}
I20260806 12:28:28.633630 31206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:28.634555 31206 log.cc:824] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:28.634776 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:28.637507 31206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124: No bootstrap required, opened a new log
I20260806 12:28:28.639530 31210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:28.637555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45757 (local address 127.28.210.126:38971)
0806 12:28:28.637685 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:28.637689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:28.637909 (+ 220us) server_negotiation.cc:407] Connection header received
0806 12:28:28.638045 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:28.638048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:28.638092 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:28.638160 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:28.638993 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:28.638997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:28.639000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 12:28:28.639013 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:28.639044 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 12:28:28.639056 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:28.639141 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:28.639233 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 12:28:28.639289 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 12:28:28.642261 31206 raft_consensus.cc:348] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63ce4f9c468421f8f97c6492622e124" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 38971 } }
I20260806 12:28:28.642622 31206 raft_consensus.cc:374] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 12:28:28.642694 31206 raft_consensus.cc:729] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63ce4f9c468421f8f97c6492622e124, State: Initialized, Role: FOLLOWER
I20260806 12:28:28.643260 31206 consensus_queue.cc:260] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63ce4f9c468421f8f97c6492622e124" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 38971 } }
I20260806 12:28:28.643455 31206 raft_consensus.cc:388] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 12:28:28.643543 31206 raft_consensus.cc:482] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 12:28:28.643672 31206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:28.645074 31206 raft_consensus.cc:504] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63ce4f9c468421f8f97c6492622e124" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 38971 } }
I20260806 12:28:28.645553 31206 leader_election.cc:302] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b63ce4f9c468421f8f97c6492622e124; no voters:
I20260806 12:28:28.645992 31206 leader_election.cc:288] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 12:28:28.646217 31219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:28.646615 31219 raft_consensus.cc:686] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [term 1 LEADER]: Becoming Leader. State: Replica: b63ce4f9c468421f8f97c6492622e124, State: Running, Role: LEADER
I20260806 12:28:28.647104 31219 consensus_queue.cc:237] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63ce4f9c468421f8f97c6492622e124" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 38971 } }
I20260806 12:28:28.647797 31206 sys_catalog.cc:564] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:28.649561 31221 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b63ce4f9c468421f8f97c6492622e124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63ce4f9c468421f8f97c6492622e124" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 38971 } } }
I20260806 12:28:28.649734 31221 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:28.650120 31222 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b63ce4f9c468421f8f97c6492622e124. Latest consensus state: current_term: 1 leader_uuid: "b63ce4f9c468421f8f97c6492622e124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63ce4f9c468421f8f97c6492622e124" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 38971 } } }
I20260806 12:28:28.650234 31222 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:28.650354 31233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:28.654348 31233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:28.660706 31233 catalog_manager.cc:1347] Generated new cluster ID: d80b086ed3fb4383a82103532e9e583d
I20260806 12:28:28.660811 31233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:28.678741 31233 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:28.679258 31233 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:28.699314 31233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b63ce4f9c468421f8f97c6492622e124: Generated new TSK 0
I20260806 12:28:28.700452 31233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:28.709524 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 31100
2026-08-06T12:28:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1657 ms)
[----------] 18 tests from AuthTokenIssuingTest (30419 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:28 dist-test-slave-lgr5 krb5kdc[31252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:28 dist-test-slave-lgr5 krb5kdc[31252](info): set up 2 sockets
Aug 06 12:28:28 dist-test-slave-lgr5 krb5kdc[31252](info): commencing operation
krb5kdc: starting...
W20260806 12:28:29.911630 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:28:30.064553 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:40327
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:40327,127.28.210.125:43765,127.28.210.124:44943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:30.200493 31272 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 12:28:30.200891 31272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:30.200932 31272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:30.201001 31272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:30.205789 31272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:30.205922 31272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:30.205961 31272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:30.205989 31272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:30.206017 31272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:30.212668 31272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:40327,127.28.210.125:43765,127.28.210.124:44943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:40327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:30.214533 31272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:30.215981 31272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:30.226155 31283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:30.226384 31272 server_base.cc:1034] running on GCE node
W20260806 12:28:30.226184 31285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:30.226184 31287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:30.227319 31272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:30.228974 31272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:30.230227 31272 hybrid_clock.cc:648] HybridClock initialized: now 1786019310230163 us; error 91 us; skew 500 ppm
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:30.234059 31272 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:28:30.235414 31272 webserver.cc:460] Webserver started at http://127.28.210.126:42725/ using document root <none> and password file <none>
I20260806 12:28:30.236171 31272 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:30.236260 31272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:30.236529 31272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:30.238963 31272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "1f742fcfe9da48ca919ef70dff27ecbd"
format_stamp: "Formatted at 2026-08-06 12:28:30 on dist-test-slave-lgr5"
server_key: "79d19c07dd1fa4436a3c7f4f2efb9235"
server_key_iv: "03d406c2249e6f33cebfe374c9fa3c00"
server_key_version: "encryptionkey@0"
I20260806 12:28:30.239602 31272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "1f742fcfe9da48ca919ef70dff27ecbd"
format_stamp: "Formatted at 2026-08-06 12:28:30 on dist-test-slave-lgr5"
server_key: "79d19c07dd1fa4436a3c7f4f2efb9235"
server_key_iv: "03d406c2249e6f33cebfe374c9fa3c00"
server_key_version: "encryptionkey@0"
I20260806 12:28:30.244410 31272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 12:28:30.247434 31300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:30.248986 31272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 12:28:30.249157 31272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "1f742fcfe9da48ca919ef70dff27ecbd"
format_stamp: "Formatted at 2026-08-06 12:28:30 on dist-test-slave-lgr5"
server_key: "79d19c07dd1fa4436a3c7f4f2efb9235"
server_key_iv: "03d406c2249e6f33cebfe374c9fa3c00"
server_key_version: "encryptionkey@0"
I20260806 12:28:30.249312 31272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:30.283367 31272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:30.287247 31272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:30.287459 31272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:30.295902 31272 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:40327
I20260806 12:28:30.296350 31378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:40327 every 8 connection(s)
I20260806 12:28:30.297503 31272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:30.301119 31380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:30.305246 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 31272
I20260806 12:28:30.305402 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:30.305663 29513 external_mini_cluster.cc:1428] Setting key 53fbb62df7358e694016556504d1b81f
I20260806 12:28:30.306425 31380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.210.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.125@KRBTEST.COM" created.
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:28:30.328228 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.309785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40327 (local address 127.0.0.1:50258)
0806 12:28:30.310488 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.310555 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:30.311769 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:30.312945 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:30.312951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:30.313418 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:30.313754 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.313770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.315652 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.315656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.316594 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.316610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.317365 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.317369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.317504 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.318189 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:30.318217 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:30.320879 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:30.323796 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.323808 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.323822 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.324211 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.324748 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.324756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.324760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.325015 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.326108 (+ 1093us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:30.326115 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:30.326278 (+ 163us) client_negotiation.cc:769] Sending connection context
0806 12:28:30.326525 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 12:28:30.327048 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":151,"thread_start_us":181,"threads_started":1}
I20260806 12:28:30.328228 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.309785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.28.210.126:40327)
0806 12:28:30.310542 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.310564 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.310705 (+ 141us) server_negotiation.cc:407] Connection header received
0806 12:28:30.312267 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.312275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.312620 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.312882 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.313966 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.315482 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.316728 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.317188 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.317369 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.321087 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.321096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.321113 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.321145 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.323611 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.324337 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.324343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.324345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.324479 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.325158 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.325162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.325163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.325886 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.326149 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.326580 (+ 431us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.326977 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":605,"spinlock_wait_cycles":29056,"thread_start_us":219,"threads_started":1}
I20260806 12:28:30.331583 31380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:28:30.332621 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.332242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:43765 (local address 127.0.0.1:56112)
0806 12:28:30.332447 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.332534 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260806 12:28:30.332857 31301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
I20260806 12:28:30.333863 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.333627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:43765 (local address 127.0.0.1:56118)
0806 12:28:30.333743 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.333792 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 12:28:30.334008 31380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.125:43765: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.125.keytab.
Entry for principal kudu/127.28.210.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.125.keytab.
I20260806 12:28:30.345674 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.125:43765
--webserver_interface=127.28.210.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.125.keytab
--principal=kudu/127.28.210.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:40327,127.28.210.125:43765,127.28.210.124:44943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:28:30.383442 31380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } attempt: 1
I20260806 12:28:30.384796 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.384373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:43765 (local address 127.0.0.1:56120)
0806 12:28:30.384625 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.384703 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 12:28:30.386045 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.385810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:43765 (local address 127.0.0.1:56132)
0806 12:28:30.385917 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.385973 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 12:28:30.386206 31380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.125:43765: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
I20260806 12:28:30.454365 31380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } attempt: 2
I20260806 12:28:30.455407 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.454989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:43765 (local address 127.0.0.1:56148)
0806 12:28:30.455221 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.455311 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260806 12:28:30.456364 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.456057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:43765 (local address 127.0.0.1:56162)
0806 12:28:30.456201 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.456256 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 12:28:30.456545 31380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.125:43765: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
W20260806 12:28:30.478394 31395 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 12:28:30.478820 31395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:30.478878 31395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:30.478960 31395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:30.483518 31395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:30.483615 31395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:30.483649 31395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:30.483676 31395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:30.483708 31395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:30.490171 31395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:40327,127.28.210.125:43765,127.28.210.124:44943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.125.keytab
--principal=kudu/127.28.210.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.125:43765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.210.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:30.491632 31395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:30.492904 31395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:30.502763 31404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:30.502753 31413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:30.502758 31406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:30.502882 31395 server_base.cc:1034] running on GCE node
I20260806 12:28:30.503829 31395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:30.505393 31395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:30.506598 31395 hybrid_clock.cc:648] HybridClock initialized: now 1786019310506572 us; error 43 us; skew 500 ppm
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:30.510011 31395 init.cc:377] Logged in from keytab as kudu/127.28.210.125@KRBTEST.COM (short username kudu)
I20260806 12:28:30.511168 31395 webserver.cc:460] Webserver started at http://127.28.210.125:37209/ using document root <none> and password file <none>
I20260806 12:28:30.511790 31395 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:30.511864 31395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:30.512108 31395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:30.514781 31395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/data/instance:
uuid: "5733a936f7304cd0b73c20423499a309"
format_stamp: "Formatted at 2026-08-06 12:28:30 on dist-test-slave-lgr5"
server_key: "77222e7a3f646fa5bde3aabcabba6087"
server_key_iv: "377e711cebbee6a6c983522aff8aec7b"
server_key_version: "encryptionkey@0"
I20260806 12:28:30.515503 31395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/wal/instance:
uuid: "5733a936f7304cd0b73c20423499a309"
format_stamp: "Formatted at 2026-08-06 12:28:30 on dist-test-slave-lgr5"
server_key: "77222e7a3f646fa5bde3aabcabba6087"
server_key_iv: "377e711cebbee6a6c983522aff8aec7b"
server_key_version: "encryptionkey@0"
I20260806 12:28:30.520673 31395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260806 12:28:30.524296 31422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:30.526073 31395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:28:30.526247 31395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/wal
uuid: "5733a936f7304cd0b73c20423499a309"
format_stamp: "Formatted at 2026-08-06 12:28:30 on dist-test-slave-lgr5"
server_key: "77222e7a3f646fa5bde3aabcabba6087"
server_key_iv: "377e711cebbee6a6c983522aff8aec7b"
server_key_version: "encryptionkey@0"
I20260806 12:28:30.526396 31395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:30.542162 31395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:30.546936 31395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:30.547194 31395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:30.547732 31380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } attempt: 3
I20260806 12:28:30.548731 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.548345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:43765 (local address 127.0.0.1:56170)
0806 12:28:30.548560 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.548633 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 12:28:30.549491 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.549265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:43765 (local address 127.0.0.1:56174)
0806 12:28:30.549376 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.549426 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 12:28:30.549609 31380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.125:43765: Network error: Client connection negotiation failed: client connection to 127.28.210.125:43765: connect: Connection refused (error 111)
I20260806 12:28:30.555671 31395 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.125:43765
I20260806 12:28:30.556116 31507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.125:43765 every 8 connection(s)
I20260806 12:28:30.556890 31395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
I20260806 12:28:30.560283 31508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:30.564855 31508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:28:30.565225 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 31395
I20260806 12:28:30.565356 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-1/wal/instance
I20260806 12:28:30.565622 29513 external_mini_cluster.cc:1428] Setting key 5d080450154e458f97c9809681904aad
WARNING: no policy specified for kudu/127.28.210.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.124@KRBTEST.COM" created.
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.125@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:28:30.587870 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.567224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.28.210.126:40327)
0806 12:28:30.567408 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.567412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.568560 (+ 1148us) server_negotiation.cc:407] Connection header received
0806 12:28:30.569653 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.569656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.569714 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.569820 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.571560 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.572290 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.574154 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.574430 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.574529 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.581520 (+ 6991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.581525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.581528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.581571 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.583658 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.585004 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.585011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.585013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.585172 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.585874 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.585880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.585881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.586323 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.586448 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.587520 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.587706 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 12:28:30.589459 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.567734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40327 (local address 127.0.0.1:50262)
0806 12:28:30.568369 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.568429 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:30.569421 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:30.570063 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:30.570073 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:30.570594 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:30.571347 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.571370 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.572446 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.572450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.573966 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.573983 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.574639 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.574644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.574902 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.575702 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:30.575729 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:30.581167 (+ 5438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:30.584056 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.584068 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.584132 (+ 64us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.584748 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.585326 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.585330 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.585332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.585628 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.586514 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:30.586526 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:30.587062 (+ 536us) client_negotiation.cc:769] Sending connection context
0806 12:28:30.587364 (+ 302us) client_negotiation.cc:240] Negotiation successful
0806 12:28:30.587904 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":150,"threads_started":1}
I20260806 12:28:30.591610 31508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.125@KRBTEST.COM for kudu/127.28.210.125@KRBTEST.COM
I20260806 12:28:30.605243 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.592310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:43765 (local address 127.0.0.1:56188)
0806 12:28:30.592521 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.592542 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:30.592674 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:30.593418 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:30.593421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:30.593677 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:30.593801 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.593808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.595086 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.595089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.595790 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.595802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.596618 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.596622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.596754 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.597360 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:30.597375 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:30.599210 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:30.602124 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.602130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.602132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.602697 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.603166 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.603170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.603173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.603304 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.604591 (+ 1287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:30.604594 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:30.604747 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 12:28:30.604883 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 12:28:30.605062 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.124.keytab.
Entry for principal kudu/127.28.210.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.124.keytab.
I20260806 12:28:30.605894 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.592449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.28.210.125:43765)
0806 12:28:30.592939 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.592950 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.592975 (+ 25us) server_negotiation.cc:407] Connection header received
0806 12:28:30.593024 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.593029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.593229 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.593379 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.593942 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.594919 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.595940 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.596431 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.596559 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.599528 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.599546 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.599563 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.599601 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.601839 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.602893 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.602900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.602901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.603030 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.603394 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.603397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.603398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.604331 (+ 933us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.604527 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.605339 (+ 812us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.605645 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":236,"threads_started":1}
I20260806 12:28:30.607012 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.124:44943
--webserver_interface=127.28.210.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.124.keytab
--principal=kudu/127.28.210.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:40327,127.28.210.125:43765,127.28.210.124:44943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:28:30.609812 31508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:28:30.610929 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.610483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42788)
0806 12:28:30.610679 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.610808 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 12:28:30.611347 31425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
I20260806 12:28:30.612792 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.612449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42790)
0806 12:28:30.612631 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.612691 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260806 12:28:30.612986 31508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:44943: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
I20260806 12:28:30.662321 31508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } attempt: 1
I20260806 12:28:30.663537 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.663050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42806)
0806 12:28:30.663271 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.663422 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 12:28:30.664531 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.664289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42808)
0806 12:28:30.664406 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.664464 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 12:28:30.664682 31508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:44943: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
I20260806 12:28:30.692845 31380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } attempt: 4
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.125@KRBTEST.COM
I20260806 12:28:30.704144 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.693555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:43765 (local address 127.0.0.1:56198)
0806 12:28:30.693759 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.693777 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:30.693906 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:30.694176 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:30.694179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:30.694398 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:30.694921 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.694931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.695830 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.695833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.696507 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.696519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.697582 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.697585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.697699 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.698477 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:30.698491 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:30.699990 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:30.702210 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.702215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.702217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.702500 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.702885 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.702888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.702890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.702992 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.703499 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:30.703502 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:30.703639 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 12:28:30.703752 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 12:28:30.703921 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 12:28:30.704178 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.693700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.28.210.125:43765)
0806 12:28:30.693886 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.693891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.693910 (+ 19us) server_negotiation.cc:407] Connection header received
0806 12:28:30.694004 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.694008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.694064 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.694133 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.695090 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.695701 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.696634 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.697428 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.697556 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.700225 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.700229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.700231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.700263 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.702050 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.702664 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.702668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.702669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.702781 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.703166 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.703169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.703171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.703404 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.703493 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.703860 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.703999 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 12:28:30.705224 31380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:28:30.706123 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.705818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42818)
0806 12:28:30.705962 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.706031 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 12:28:30.706912 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.706652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42828)
0806 12:28:30.706787 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.706846 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 12:28:30.707050 31380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:44943: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
I20260806 12:28:30.732844 31508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } attempt: 2
I20260806 12:28:30.733850 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.733457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42844)
0806 12:28:30.733691 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.733757 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260806 12:28:30.734810 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.734556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42854)
0806 12:28:30.734690 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.734741 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 12:28:30.734951 31508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:44943: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
W20260806 12:28:30.753513 31516 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 12:28:30.753918 31516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:30.753971 31516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:30.754045 31516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:30.758741 31516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:30.758867 31516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:30.758899 31516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:30.758929 31516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:30.758961 31516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:30.765195 31516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:40327,127.28.210.125:43765,127.28.210.124:44943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.124.keytab
--principal=kudu/127.28.210.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.124:44943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.210.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:30.766219 31380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } attempt: 1
I20260806 12:28:30.766695 31516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:30.767186 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.766878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42866)
0806 12:28:30.767029 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.767097 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 12:28:30.767936 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.767681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42878)
0806 12:28:30.767809 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.767860 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 12:28:30.768114 31380 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:44943: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
I20260806 12:28:30.768317 31516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:30.779574 31525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:30.779639 31516 server_base.cc:1034] running on GCE node
W20260806 12:28:30.779574 31529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:30.779577 31527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:30.780557 31516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:30.781937 31516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:30.783172 31516 hybrid_clock.cc:648] HybridClock initialized: now 1786019310783148 us; error 42 us; skew 500 ppm
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:30.787007 31516 init.cc:377] Logged in from keytab as kudu/127.28.210.124@KRBTEST.COM (short username kudu)
I20260806 12:28:30.788398 31516 webserver.cc:460] Webserver started at http://127.28.210.124:37631/ using document root <none> and password file <none>
I20260806 12:28:30.789151 31516 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:30.789255 31516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:30.789572 31516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:30.792438 31516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/data/instance:
uuid: "2badf5e0fd8a460d8c7158a30b728f9e"
format_stamp: "Formatted at 2026-08-06 12:28:30 on dist-test-slave-lgr5"
server_key: "e92aa00bbf392541ab36735baab45e4d"
server_key_iv: "c4acd8087c2a3e5ba175b8f06a30cbc6"
server_key_version: "encryptionkey@0"
I20260806 12:28:30.793236 31516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/wal/instance:
uuid: "2badf5e0fd8a460d8c7158a30b728f9e"
format_stamp: "Formatted at 2026-08-06 12:28:30 on dist-test-slave-lgr5"
server_key: "e92aa00bbf392541ab36735baab45e4d"
server_key_iv: "c4acd8087c2a3e5ba175b8f06a30cbc6"
server_key_version: "encryptionkey@0"
I20260806 12:28:30.798415 31516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 12:28:30.801626 31544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:30.803216 31516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 12:28:30.803399 31516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/wal
uuid: "2badf5e0fd8a460d8c7158a30b728f9e"
format_stamp: "Formatted at 2026-08-06 12:28:30 on dist-test-slave-lgr5"
server_key: "e92aa00bbf392541ab36735baab45e4d"
server_key_iv: "c4acd8087c2a3e5ba175b8f06a30cbc6"
server_key_version: "encryptionkey@0"
I20260806 12:28:30.803546 31516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:30.824584 31516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:30.826145 31508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } attempt: 3
I20260806 12:28:30.827005 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.826711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42886)
0806 12:28:30.826855 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.826915 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 12:28:30.827792 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.827522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42890)
0806 12:28:30.827678 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.827728 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260806 12:28:30.827961 31508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:44943: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
I20260806 12:28:30.828570 31516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:30.828747 31516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:30.835290 31380 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } attempt: 2
I20260806 12:28:30.836258 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.835878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42894)
0806 12:28:30.836052 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.836151 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 12:28:30.836767 31516 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.124:44943
I20260806 12:28:30.837090 31631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.124:44943 every 8 connection(s)
I20260806 12:28:30.837963 31516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
I20260806 12:28:30.838943 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 31516
I20260806 12:28:30.839072 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786018995816160-29513-0/minicluster-data/master-2/wal/instance
I20260806 12:28:30.839380 29513 external_mini_cluster.cc:1428] Setting key c3008a2195130f6b811c5971809e7467
I20260806 12:28:30.841668 31633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.124@KRBTEST.COM
I20260806 12:28:30.846817 31633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.124@KRBTEST.COM
I20260806 12:28:30.852643 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.836950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42896)
0806 12:28:30.837084 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.837100 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:30.837215 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:30.840257 (+ 3042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:30.840261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:30.840480 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:30.840629 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.840637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.843097 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.843101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.843920 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.843933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.844890 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.844894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.844991 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.845562 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:30.845576 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:30.846979 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:30.849546 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.849549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.849552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.849841 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.850383 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.850387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.850389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.850504 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.851304 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:30.851309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:30.851421 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 12:28:30.851514 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 12:28:30.852459 (+ 945us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":764}
I20260806 12:28:30.853219 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.840115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.28.210.124:44943)
0806 12:28:30.840413 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.840421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.840666 (+ 245us) server_negotiation.cc:407] Connection header received
0806 12:28:30.840813 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.840816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.840861 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.840962 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.841876 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.842936 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.843898 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.844734 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.844962 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.848112 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.848117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.848120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.848155 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.849849 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.850400 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.850404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.850405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.850499 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.850801 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.850804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.850806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.851154 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.851363 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.851672 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.852050 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":280,"server-negotiator.queue_time_us":232,"spinlock_wait_cycles":6016,"thread_start_us":97,"threads_started":1}
I20260806 12:28:30.853529 31632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.838025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.28.210.124:44943)
0806 12:28:30.838547 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.838569 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.838613 (+ 44us) server_negotiation.cc:407] Connection header received
0806 12:28:30.839394 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.839414 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.839838 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.840859 (+ 1021us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.841320 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.842936 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.844048 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.844734 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.844962 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.847225 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.847233 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.847254 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.847291 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.849396 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.849976 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.849981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.849988 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.850179 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.850604 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.850608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.850609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.851159 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.851615 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.851690 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.852142 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":236,"server-negotiator.queue_time_us":348,"spinlock_wait_cycles":4992,"thread_start_us":147,"threads_started":1}
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.124@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:28:30.858716 31380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd: Bootstrap starting.
I20260806 12:28:30.861608 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.850298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.28.210.126:40327)
0806 12:28:30.850442 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.850446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.850457 (+ 11us) server_negotiation.cc:407] Connection header received
0806 12:28:30.850488 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.850490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.850526 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.850595 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.851486 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.852244 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.853686 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.853999 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.854075 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.857190 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.857194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.857197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.857230 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.858753 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.859539 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.859542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.859543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.859645 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.860047 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.860051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.860052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.860462 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.860582 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.861110 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.861260 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":38}
I20260806 12:28:30.862378 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.849205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:40327 (local address 127.0.0.1:50272)
0806 12:28:30.849739 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.849801 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:30.849933 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:30.850746 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:30.850752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:30.851205 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:30.851373 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.851382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.852365 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.852369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.853520 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.853537 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.854225 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.854227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.854342 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.854928 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:30.854945 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:30.856987 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:30.858909 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.858926 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.858933 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.859316 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.859759 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.859762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.859764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.859888 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.860660 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:30.860669 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:30.860831 (+ 162us) client_negotiation.cc:769] Sending connection context
0806 12:28:30.861916 (+ 1085us) client_negotiation.cc:240] Negotiation successful
0806 12:28:30.862212 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":177,"threads_started":1}
I20260806 12:28:30.862423 31380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:30.863638 31380 log.cc:824] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:30.863705 31633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:28:30.866348 31380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd: No bootstrap required, opened a new log
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.124@KRBTEST.COM for kudu/127.28.210.125@KRBTEST.COM
I20260806 12:28:30.871122 31380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } }
I20260806 12:28:30.871423 31380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f742fcfe9da48ca919ef70dff27ecbd, State: Initialized, Role: FOLLOWER
I20260806 12:28:30.871804 31380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } }
I20260806 12:28:30.872051 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.860267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.28.210.126:40327)
0806 12:28:30.860716 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.860732 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.860749 (+ 17us) server_negotiation.cc:407] Connection header received
0806 12:28:30.860789 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.860792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.860856 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.860967 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.862166 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.863527 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.864670 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.865014 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.865109 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.867909 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.867913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.867915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.867940 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.869466 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.869962 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.869965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.869967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.870068 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.870428 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.870433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.870435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.870677 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.870884 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.871620 (+ 736us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.871755 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":123,"threads_started":1}
I20260806 12:28:30.873682 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.864395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:43765 (local address 127.0.0.1:56212)
0806 12:28:30.864593 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.864609 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:30.864703 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:30.865032 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:30.865035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:30.865241 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:30.865382 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.865390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.866367 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.866371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.867130 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.867145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.867908 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.867913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.868004 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.868671 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:30.868689 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:30.870350 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:30.871993 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.871996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.871998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.872280 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.872589 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.872592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.872594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.872678 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.873122 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:30.873125 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:30.873236 (+ 111us) client_negotiation.cc:769] Sending connection context
0806 12:28:30.873362 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 12:28:30.873523 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260806 12:28:30.873680 31380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:30.873872 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.864512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.28.210.125:43765)
0806 12:28:30.864637 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.864642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.864658 (+ 16us) server_negotiation.cc:407] Connection header received
0806 12:28:30.864829 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.864833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.864901 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.864990 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.865552 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.866227 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.867317 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.867642 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.867757 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.870506 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.870509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.870511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.870539 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.871827 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.872394 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.872398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.872400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.872493 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.872785 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.872788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.872790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.873007 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.873109 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.873598 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.873718 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 12:28:30.874459 31633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:28:30.876220 31645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } } }
I20260806 12:28:30.876770 31645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [sys.catalog]: This master's current role is: FOLLOWER
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.124@KRBTEST.COM for kudu/127.28.210.124@KRBTEST.COM
W20260806 12:28:30.881999 31662 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:28:30.882098 31662 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:28:30.883963 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.874969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42916)
0806 12:28:30.875120 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.875136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:30.875212 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:30.875588 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:30.875591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:30.875757 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:30.875843 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.875850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.876802 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.876804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.877488 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.877502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.878058 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.878061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.878143 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.878697 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:30.878712 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:30.880372 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:30.882386 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.882390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.882392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.882593 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.882875 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.882878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.882880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.882960 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.883394 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:30.883397 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:30.883499 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 12:28:30.883593 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 12:28:30.883763 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 12:28:30.883963 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.875204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.28.210.124:44943)
0806 12:28:30.875309 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.875314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.875327 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:28:30.875373 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.875376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.875416 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.875491 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.875979 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.876680 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.877618 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.877932 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.878035 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.880529 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.880533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.880535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.880560 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.882270 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.882703 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.882707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.882709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.882788 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.883042 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.883045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.883047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.883295 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.883393 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.883653 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.883823 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":39}
I20260806 12:28:30.880465 31272 master_runner.cc:186] Error getting master registration for 127.28.210.125:43765: OK, instance_id { permanent_uuid: "5733a936f7304cd0b73c20423499a309" instance_seqno: 1786019310553298 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 12:28:30.884707 31272 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.125@KRBTEST.COM
I20260806 12:28:30.887104 31633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e: Bootstrap starting.
I20260806 12:28:30.889870 31633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:30.890936 31633 log.cc:824] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:30.891969 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.873273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.28.210.125:43765)
0806 12:28:30.873622 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.873631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.874219 (+ 588us) server_negotiation.cc:407] Connection header received
0806 12:28:30.874261 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.874264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.874310 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.874392 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.875605 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.876472 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.877672 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.877936 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.878032 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.887627 (+ 9595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.887632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.887635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.887676 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.889773 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.890413 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.890417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.890419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.890525 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.890900 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.890904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.890906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.891163 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.891381 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.891627 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.891768 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":139,"threads_started":1}
I20260806 12:28:30.893778 31633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e: No bootstrap required, opened a new log
I20260806 12:28:30.896399 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:30.898047 31633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } }
I20260806 12:28:30.898085 31341 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1f742fcfe9da48ca919ef70dff27ecbd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50280
I20260806 12:28:30.898280 31633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2badf5e0fd8a460d8c7158a30b728f9e, State: Initialized, Role: FOLLOWER
I20260806 12:28:30.898471 31341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 12:28:30.898562 31633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } }
I20260806 12:28:30.898590 31341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:30.899067 31665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } } }
I20260806 12:28:30.899202 31665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:28:30.900008 31341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } }
I20260806 12:28:30.900170 31633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:30.902649 31341 leader_election.cc:288] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [CANDIDATE]: Term 1 election: Requested vote from peers 5733a936f7304cd0b73c20423499a309 (127.28.210.125:43765), 2badf5e0fd8a460d8c7158a30b728f9e (127.28.210.124:44943)
I20260806 12:28:30.902470 31592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2badf5e0fd8a460d8c7158a30b728f9e"
I20260806 12:28:30.903020 31592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:30.904460 31592 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f742fcfe9da48ca919ef70dff27ecbd in term 1.
I20260806 12:28:30.905222 31305 leader_election.cc:302] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f742fcfe9da48ca919ef70dff27ecbd, 2badf5e0fd8a460d8c7158a30b728f9e; no voters:
I20260806 12:28:30.905558 31645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:30.905757 31645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [term 1 LEADER]: Becoming Leader. State: Replica: 1f742fcfe9da48ca919ef70dff27ecbd, State: Running, Role: LEADER
W20260806 12:28:30.905833 31685 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:28:30.905908 31685 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:28:30.906112 31645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } }
I20260806 12:28:30.908103 31645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f742fcfe9da48ca919ef70dff27ecbd. Latest consensus state: current_term: 1 leader_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } } }
I20260806 12:28:30.908229 31645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:30.912729 31689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:30.914974 31689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:30.912775 31516 master_runner.cc:186] Error getting master registration for 127.28.210.125:43765: OK, instance_id { permanent_uuid: "5733a936f7304cd0b73c20423499a309" instance_seqno: 1786019310553298 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 12:28:30.917300 31516 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 12:28:30.928544 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.910577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42932 (local address 127.28.210.124:44943)
0806 12:28:30.910704 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.910709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.917877 (+ 7168us) server_negotiation.cc:407] Connection header received
0806 12:28:30.918134 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.918137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.918195 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.918248 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.919484 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.920366 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.921709 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.922213 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.922324 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.924576 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.924580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.924583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.924619 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.926455 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.926987 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.926991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.926993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.927094 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.927427 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.927431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.927433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.927679 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.927779 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.928214 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.928351 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 12:28:30.931222 31592 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [term 1 FOLLOWER]: Refusing update from remote peer 1f742fcfe9da48ca919ef70dff27ecbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:28:30.932073 31687 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:28:30.937613 31665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f742fcfe9da48ca919ef70dff27ecbd. Latest consensus state: current_term: 1 leader_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } } }
I20260806 12:28:30.937775 31665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:28:30.942502 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.911379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.28.210.126:40327)
0806 12:28:30.911493 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.911498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.920443 (+ 8945us) server_negotiation.cc:407] Connection header received
0806 12:28:30.920656 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.920660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.920724 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.920784 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.921868 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.922816 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.925689 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.926110 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.926211 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.930487 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.930492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.930494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.930525 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.932297 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.934225 (+ 1928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.934229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.934231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.934350 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.940836 (+ 6486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.940842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.940844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.941223 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.941431 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.942098 (+ 667us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.942298 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 12:28:30.942828 31687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } } }
I20260806 12:28:30.943009 31687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:30.945683 31665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } } }
I20260806 12:28:30.945890 31665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:28:30.947091 31707 mvcc.cc:205] Tried to move back new op lower bound from 7315535097560907776 to 7315535097475051520. Current Snapshot: MvccSnapshot[applied={T|T < 7315535097560907776}]
I20260806 12:28:30.949285 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.912163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.28.210.125:43765)
0806 12:28:30.912292 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.912296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.924423 (+ 12127us) server_negotiation.cc:407] Connection header received
0806 12:28:30.924630 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.924633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.924694 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.924751 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.927022 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.927820 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.930447 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.930841 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.930952 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.934236 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.934241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.934244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.934274 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.935916 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.941017 (+ 5101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.941022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.941024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.941165 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.947956 (+ 6791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.947962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.947965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.948309 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.948471 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.948901 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.949087 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 12:28:30.952320 31689 catalog_manager.cc:1347] Generated new cluster ID: b9d2f158d46e40d987292996435ed578
I20260806 12:28:30.952435 31689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 12:28:30.971128 31508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } attempt: 4
Aug 06 12:28:30 dist-test-slave-lgr5 krb5kdc[31252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019310, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.125@KRBTEST.COM for kudu/127.28.210.124@KRBTEST.COM
I20260806 12:28:30.982051 31689 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:30.985545 31689 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:30.989282 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.971868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:44943 (local address 127.0.0.1:42934)
0806 12:28:30.972213 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:30.972231 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:30.972328 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:30.973000 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:30.973003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:30.973257 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:30.973382 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.973390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.974447 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.974451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.975240 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:30.975255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.975862 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.975866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:30.975952 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.976771 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:30.976788 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:30.979691 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:30.987074 (+ 7383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.987078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.987081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.987466 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.987856 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:30.987859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:30.987862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:30.987987 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:30.988641 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:30.988645 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:30.988786 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 12:28:30.988883 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 12:28:30.989084 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243}
I20260806 12:28:30.990264 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:30.972520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.28.210.124:44943)
0806 12:28:30.972664 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:30.972669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:30.972682 (+ 13us) server_negotiation.cc:407] Connection header received
0806 12:28:30.972727 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:30.972730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:30.972781 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:30.972859 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:30.973522 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.974302 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.975385 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:30.975742 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:30.975846 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:30.979915 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:30.979920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:30.979923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:30.979958 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:30.981907 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.987614 (+ 5707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.987619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.987621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.987740 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:30.988136 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:30.988140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:30.988142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:30.988426 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:30.988550 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:30.989883 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0806 12:28:30.990078 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 12:28:30.994529 31508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309: Bootstrap starting.
I20260806 12:28:30.998320 31508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:30.999635 31508 log.cc:824] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:31.015064 31508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309: No bootstrap required, opened a new log
I20260806 12:28:31.018491 31689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd: Generated new TSK 0
I20260806 12:28:31.019548 31689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 12:28:31.021059 31508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } }
I20260806 12:28:31.021494 31508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5733a936f7304cd0b73c20423499a309, State: Initialized, Role: FOLLOWER
I20260806 12:28:31.021950 31508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } }
I20260806 12:28:31.024025 31508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:31.024444 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5733a936f7304cd0b73c20423499a309"
I20260806 12:28:31.024729 31464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:31.026952 31464 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f742fcfe9da48ca919ef70dff27ecbd in term 1.
I20260806 12:28:31.027487 31715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } } }
I20260806 12:28:31.028277 31462 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [term 1 FOLLOWER]: Refusing update from remote peer 1f742fcfe9da48ca919ef70dff27ecbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:28:31.028333 31715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:28:31.029014 31645 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 12:28:31.031899 31732 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:28:31.032073 31732 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:28:31.043299 31715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f742fcfe9da48ca919ef70dff27ecbd. Latest consensus state: current_term: 1 leader_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } } }
I20260806 12:28:31.043533 31715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:28:31.043651 31715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } } }
I20260806 12:28:31.043731 31715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 12:28:31.049319 31431 tablet.cc:2369] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:28:31.051584 31735 mvcc.cc:205] Tried to move back new op lower bound from 7315535097910951936 to 7315535097475051520. Current Snapshot: MvccSnapshot[applied={T|T < 7315535097560907776}]
W20260806 12:28:31.074349 31635 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.070368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.28.210.124:44943)
0806 12:28:31.070513 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.070518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.072902 (+ 2384us) server_negotiation.cc:407] Connection header received
0806 12:28:31.073157 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.073161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.073223 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.073291 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.074177 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42946: BlockingRecv error: recv got EOF from 127.0.0.1:42946 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 12:28:31.076859 31639 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.071372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.28.210.126:40327)
0806 12:28:31.071500 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.071505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.075370 (+ 3865us) server_negotiation.cc:407] Connection header received
0806 12:28:31.075582 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.075585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.075644 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.075698 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.076695 (+ 997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50298: BlockingRecv error: recv got EOF from 127.0.0.1:50298 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 12:28:31.078750 31647 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.072213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.28.210.125:43765)
0806 12:28:31.072340 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.072345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.076279 (+ 3934us) server_negotiation.cc:407] Connection header received
0806 12:28:31.077697 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.077701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.077764 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.077832 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.078601 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56250: BlockingRecv error: recv got EOF from 127.0.0.1:56250 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 12:28:31.085433 31647 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.083004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.28.210.125:43765)
0806 12:28:31.083160 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.083165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.083938 (+ 773us) server_negotiation.cc:407] Connection header received
0806 12:28:31.084257 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.084261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.084488 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.084566 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.085286 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56264: BlockingRecv error: recv got EOF from 127.0.0.1:56264 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 12:28:31.091630 31647 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.089876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.28.210.125:43765)
0806 12:28:31.090003 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.090008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.090574 (+ 566us) server_negotiation.cc:407] Connection header received
0806 12:28:31.090785 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.090789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.090857 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.090909 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.091501 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56272: BlockingRecv error: recv got EOF from 127.0.0.1:56272 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 12:28:31.098990 31647 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.097115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.28.210.125:43765)
0806 12:28:31.097251 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.097257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.097869 (+ 612us) server_negotiation.cc:407] Connection header received
0806 12:28:31.098108 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.098112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.098171 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.098225 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.098847 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56284: BlockingRecv error: recv got EOF from 127.0.0.1:56284 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 12:28:31.108527 31647 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.106437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.28.210.125:43765)
0806 12:28:31.106577 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.106582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.107235 (+ 653us) server_negotiation.cc:407] Connection header received
0806 12:28:31.107459 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.107463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.107542 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.107597 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.108392 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56296: BlockingRecv error: recv got EOF from 127.0.0.1:56296 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 12:28:31.125010 31647 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.121616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.28.210.125:43765)
0806 12:28:31.121764 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.121770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.122638 (+ 868us) server_negotiation.cc:407] Connection header received
0806 12:28:31.123374 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.123379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.123454 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.123516 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.124459 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56302: BlockingRecv error: recv got EOF from 127.0.0.1:56302 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 12:28:31.147828 31647 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.146581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.28.210.125:43765)
0806 12:28:31.146746 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.146752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.146771 (+ 19us) server_negotiation.cc:407] Connection header received
0806 12:28:31.146993 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.146996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.147058 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.147151 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.147689 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56318: BlockingRecv error: recv got EOF from 127.0.0.1:56318 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 12:28:31.184478 31647 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.182904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.28.210.125:43765)
0806 12:28:31.183076 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.183082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.183304 (+ 222us) server_negotiation.cc:407] Connection header received
0806 12:28:31.183475 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.183478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.183543 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.183642 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.184332 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56324: BlockingRecv error: recv got EOF from 127.0.0.1:56324 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260806 12:28:31.252346 31647 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.251098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.28.210.125:43765)
0806 12:28:31.251266 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.251271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.251367 (+ 96us) server_negotiation.cc:407] Connection header received
0806 12:28:31.251554 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.251557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.251611 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.251687 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.252213 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56332: BlockingRecv error: recv got EOF from 127.0.0.1:56332 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 12:28:31.384652 31647 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.383181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.28.210.125:43765)
0806 12:28:31.383531 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.383535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.383552 (+ 17us) server_negotiation.cc:407] Connection header received
0806 12:28:31.383616 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.383620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.383671 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.383759 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.384506 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56342: BlockingRecv error: recv got EOF from 127.0.0.1:56342 (error 108)
Metrics: {"server-negotiator.queue_time_us":238}
I20260806 12:28:31.615108 31828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f742fcfe9da48ca919ef70dff27ecbd" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 40327 } } peers { permanent_uuid: "5733a936f7304cd0b73c20423499a309" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 43765 } } peers { permanent_uuid: "2badf5e0fd8a460d8c7158a30b728f9e" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 44943 } } }
I20260806 12:28:31.615523 31828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f742fcfe9da48ca919ef70dff27ecbd [sys.catalog]: This master's current role is: LEADER
W20260806 12:28:31.645792 31647 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:31.643813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.28.210.125:43765)
0806 12:28:31.644178 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:31.644185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:31.644606 (+ 421us) server_negotiation.cc:407] Connection header received
0806 12:28:31.644748 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:31.644751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:31.644823 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:31.644927 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:31.645650 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56348: BlockingRecv error: recv got EOF from 127.0.0.1:56348 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
I20260806 12:28:31.907511 31685 catalog_manager.cc:1259] Loaded cluster ID: b9d2f158d46e40d987292996435ed578
I20260806 12:28:31.907599 31685 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e: loading cluster ID for follower catalog manager: success
I20260806 12:28:31.909430 31685 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e: acquiring CA information for follower catalog manager: success
I20260806 12:28:31.910718 31685 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2badf5e0fd8a460d8c7158a30b728f9e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 12:28:32.033622 31732 catalog_manager.cc:1259] Loaded cluster ID: b9d2f158d46e40d987292996435ed578
I20260806 12:28:32.033720 31732 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309: loading cluster ID for follower catalog manager: success
I20260806 12:28:32.035553 31732 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309: acquiring CA information for follower catalog manager: success
I20260806 12:28:32.037120 31732 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5733a936f7304cd0b73c20423499a309: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 12:28:32.164721 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:32.160336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.28.210.125:43765)
0806 12:28:32.160631 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:32.160637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:32.160750 (+ 113us) server_negotiation.cc:407] Connection header received
0806 12:28:32.160916 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:32.160919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:32.161088 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:32.161203 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:28:32.161491 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:32.162048 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:32.162940 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:32.163180 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:32.163276 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:32.163627 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:28:32.164188 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:28:32.164303 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:32.164379 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 12:28:32.164446 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260806 12:28:32.172171 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:32.168412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.28.210.124:44943)
0806 12:28:32.168660 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:32.168664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:32.168679 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:28:32.168774 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:32.168777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:32.168914 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:32.169023 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:28:32.169200 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:32.169761 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:32.170553 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:32.170807 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:32.170902 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:32.171151 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:28:32.171659 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:28:32.171763 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:32.171888 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 12:28:32.171955 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":83,"threads_started":1}
I20260806 12:28:32.180269 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:32.175769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.28.210.124:44943)
0806 12:28:32.175922 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:32.175944 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:32.176276 (+ 332us) server_negotiation.cc:407] Connection header received
0806 12:28:32.176496 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:32.176499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:32.176665 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:32.176944 (+ 279us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:28:32.177554 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:32.178167 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:32.179111 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:32.179336 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:32.179430 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:32.179740 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:28:32.179825 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:28:32.179901 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:32.180023 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 12:28:32.180106 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 12:28:32.181274 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:32.175690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.28.210.126:40327)
0806 12:28:32.175952 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:32.175956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:32.176246 (+ 290us) server_negotiation.cc:407] Connection header received
0806 12:28:32.176557 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:32.176559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:32.176696 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:32.177025 (+ 329us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:28:32.177582 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:32.178378 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:32.179632 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:32.179986 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:32.180109 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:32.180517 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:28:32.180916 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:28:32.181002 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:32.181055 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 12:28:32.181118 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20260806 12:28:32.181521 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:32.175758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.28.210.125:43765)
0806 12:28:32.175947 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:32.175951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:32.176399 (+ 448us) server_negotiation.cc:407] Connection header received
0806 12:28:32.176492 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:32.176595 (+ 103us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:32.176662 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:32.176961 (+ 299us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:28:32.177399 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:32.178253 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:32.179736 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:32.180053 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:32.180179 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:32.180990 (+ 811us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:28:32.181077 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:28:32.181148 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:32.181297 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 12:28:32.181372 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 12:28:32.184828 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 31272
I20260806 12:28:32.200616 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 31395
I20260806 12:28:32.211253 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 31516
2026-08-06T12:28:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3496 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 12:28:32 dist-test-slave-lgr5 krb5kdc[31867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 12:28:32 dist-test-slave-lgr5 krb5kdc[31867](info): set up 2 sockets
Aug 06 12:28:32 dist-test-slave-lgr5 krb5kdc[31867](info): commencing operation
krb5kdc: starting...
W20260806 12:28:33.413962 29513 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/test-user.keytab.
Aug 06 12:28:33 dist-test-slave-lgr5 krb5kdc[31867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T12:28:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T12:28:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
Entry for principal kudu/127.28.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab.
I20260806 12:28:33.563031 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34209
--webserver_interface=127.28.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:34209,127.28.210.125:34105,127.28.210.124:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:33.687801 31887 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 12:28:33.688236 31887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:33.688279 31887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:33.688352 31887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:33.692739 31887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:33.692816 31887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:33.692843 31887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:33.692866 31887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:33.692896 31887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:33.698832 31887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:34209,127.28.210.125:34105,127.28.210.124:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.126.keytab
--principal=kudu/127.28.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.126:34209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:33.700160 31887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:33.701315 31887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:33.710312 31894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:33.710307 31897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:33.710566 31887 server_base.cc:1034] running on GCE node
W20260806 12:28:33.710685 31895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:33.711246 31887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:33.712582 31887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:33.713766 31887 hybrid_clock.cc:648] HybridClock initialized: now 1786019313713746 us; error 47 us; skew 500 ppm
Aug 06 12:28:33 dist-test-slave-lgr5 krb5kdc[31867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019313, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:33.722308 31887 init.cc:377] Logged in from keytab as kudu/127.28.210.126@KRBTEST.COM (short username kudu)
I20260806 12:28:33.723587 31887 webserver.cc:460] Webserver started at http://127.28.210.126:35709/ using document root <none> and password file <none>
I20260806 12:28:33.724308 31887 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:33.724392 31887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:33.724647 31887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:33.727097 31887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/data/instance:
uuid: "d46b26dc456c41e1bafe4c18e1212da8"
format_stamp: "Formatted at 2026-08-06 12:28:33 on dist-test-slave-lgr5"
server_key: "52dfe9049889ad85dff7b841095351ad"
server_key_iv: "947e3927ed109a1b0fc44e6b975198d4"
server_key_version: "encryptionkey@0"
I20260806 12:28:33.727780 31887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/wal/instance:
uuid: "d46b26dc456c41e1bafe4c18e1212da8"
format_stamp: "Formatted at 2026-08-06 12:28:33 on dist-test-slave-lgr5"
server_key: "52dfe9049889ad85dff7b841095351ad"
server_key_iv: "947e3927ed109a1b0fc44e6b975198d4"
server_key_version: "encryptionkey@0"
I20260806 12:28:33.733099 31887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260806 12:28:33.736943 31911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:33.738476 31887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 12:28:33.738641 31887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/wal
uuid: "d46b26dc456c41e1bafe4c18e1212da8"
format_stamp: "Formatted at 2026-08-06 12:28:33 on dist-test-slave-lgr5"
server_key: "52dfe9049889ad85dff7b841095351ad"
server_key_iv: "947e3927ed109a1b0fc44e6b975198d4"
server_key_version: "encryptionkey@0"
I20260806 12:28:33.738822 31887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:33.754594 31887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:33.758118 31887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:33.758325 31887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:33.766216 31887 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.126:34209
I20260806 12:28:33.766500 31987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.126:34209 every 8 connection(s)
I20260806 12:28:33.767309 31887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/data/info.pb
I20260806 12:28:33.770409 31988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:33.771622 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 31887
I20260806 12:28:33.771776 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-0/wal/instance
I20260806 12:28:33.772154 29513 external_mini_cluster.cc:1428] Setting key 78f5c32eb2a387aff5dd926b23797b87
I20260806 12:28:33.775429 31988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.210.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.125@KRBTEST.COM" created.
Aug 06 12:28:33 dist-test-slave-lgr5 krb5kdc[31867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019313, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:28:33.792975 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:33.778217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34209 (local address 127.0.0.1:58124)
0806 12:28:33.778890 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:33.778932 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:33.779900 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:33.780686 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:33.780693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:33.781162 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:33.781464 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:33.781485 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:33.782735 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:33.782739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:33.783511 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:33.783523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:33.784337 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:33.784342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:33.784519 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:33.785208 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:33.785228 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:33.787028 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:33.789330 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:33.789341 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:33.789354 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:33.789619 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:33.790088 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:33.790092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:33.790095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:33.790285 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:33.791093 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:33.791101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:33.791320 (+ 219us) client_negotiation.cc:769] Sending connection context
0806 12:28:33.791574 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 12:28:33.792107 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":102,"spinlock_wait_cycles":113024,"thread_start_us":279,"threads_started":1}
I20260806 12:28:33.792989 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:33.778217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.28.210.126:34209)
0806 12:28:33.778745 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:33.778767 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:33.779056 (+ 289us) server_negotiation.cc:407] Connection header received
0806 12:28:33.780257 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:33.780261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:33.780485 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:33.780673 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:33.781638 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:33.782607 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:33.783643 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:33.784167 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:33.784356 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:33.787235 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:33.787246 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:33.787263 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:33.787295 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:33.789188 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:33.789746 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:33.789753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:33.789756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:33.789915 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:33.790411 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:33.790414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:33.790415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:33.790926 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:33.791050 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:33.791678 (+ 628us) server_negotiation.cc:299] Negotiation successful
0806 12:28:33.791945 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":155,"threads_started":1}
I20260806 12:28:33.796020 31988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:28:33.797130 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:33.796705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:34105 (local address 127.0.0.1:34712)
0806 12:28:33.796873 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:33.797010 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 12:28:33.797415 31914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.210.125:34105: connect: Connection refused (error 111)
I20260806 12:28:33.798628 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:33.798236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:34105 (local address 127.0.0.1:34724)
0806 12:28:33.798386 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:33.798484 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 12:28:33.798820 31988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.125:34105: Network error: Client connection negotiation failed: client connection to 127.28.210.125:34105: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.125.keytab.
Entry for principal kudu/127.28.210.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.125.keytab.
I20260806 12:28:33.804703 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.125:34105
--webserver_interface=127.28.210.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.125.keytab
--principal=kudu/127.28.210.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:34209,127.28.210.125:34105,127.28.210.124:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 12:28:33.848193 31988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } attempt: 1
I20260806 12:28:33.849377 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:33.848919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:34105 (local address 127.0.0.1:34726)
0806 12:28:33.849144 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:33.849248 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 12:28:33.850402 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:33.850083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:34105 (local address 127.0.0.1:34732)
0806 12:28:33.850234 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:33.850312 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 12:28:33.850558 31988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.125:34105: Network error: Client connection negotiation failed: client connection to 127.28.210.125:34105: connect: Connection refused (error 111)
I20260806 12:28:33.918778 31988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } attempt: 2
I20260806 12:28:33.919854 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:33.919426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:34105 (local address 127.0.0.1:34734)
0806 12:28:33.919659 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:33.919745 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 12:28:33.920837 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:33.920541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:34105 (local address 127.0.0.1:34736)
0806 12:28:33.920686 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:33.920750 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.125:34105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 12:28:33.921020 31988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.125:34105: Network error: Client connection negotiation failed: client connection to 127.28.210.125:34105: connect: Connection refused (error 111)
W20260806 12:28:33.932413 31996 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 12:28:33.932765 31996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:33.932807 31996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:33.932875 31996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:33.937291 31996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:33.937382 31996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:33.937400 31996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:33.937422 31996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:33.937438 31996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:33.943498 31996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:34209,127.28.210.125:34105,127.28.210.124:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.125.keytab
--principal=kudu/127.28.210.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.125:34105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.210.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:33.944785 31996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:33.945945 31996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 12:28:33.956069 31996 server_base.cc:1034] running on GCE node
W20260806 12:28:33.955998 32003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:33.956007 32002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:33.955998 32006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:33.957125 31996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:33.958696 31996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:33.960136 31996 hybrid_clock.cc:648] HybridClock initialized: now 1786019313960007 us; error 117 us; skew 500 ppm
Aug 06 12:28:33 dist-test-slave-lgr5 krb5kdc[31867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019313, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:33.963959 31996 init.cc:377] Logged in from keytab as kudu/127.28.210.125@KRBTEST.COM (short username kudu)
I20260806 12:28:33.965335 31996 webserver.cc:460] Webserver started at http://127.28.210.125:36193/ using document root <none> and password file <none>
I20260806 12:28:33.966039 31996 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:33.966118 31996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:33.966367 31996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:33.968984 31996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/data/instance:
uuid: "1d1611022b894c71ab6ec11d05dae4d0"
format_stamp: "Formatted at 2026-08-06 12:28:33 on dist-test-slave-lgr5"
server_key: "7448c694f629b63286ea96d7da83c56e"
server_key_iv: "5fe6c72250b06061461c59229689deb8"
server_key_version: "encryptionkey@0"
I20260806 12:28:33.969697 31996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/wal/instance:
uuid: "1d1611022b894c71ab6ec11d05dae4d0"
format_stamp: "Formatted at 2026-08-06 12:28:33 on dist-test-slave-lgr5"
server_key: "7448c694f629b63286ea96d7da83c56e"
server_key_iv: "5fe6c72250b06061461c59229689deb8"
server_key_version: "encryptionkey@0"
I20260806 12:28:33.975206 31996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 12:28:33.978735 32015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:33.980269 31996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 12:28:33.980456 31996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/wal
uuid: "1d1611022b894c71ab6ec11d05dae4d0"
format_stamp: "Formatted at 2026-08-06 12:28:33 on dist-test-slave-lgr5"
server_key: "7448c694f629b63286ea96d7da83c56e"
server_key_iv: "5fe6c72250b06061461c59229689deb8"
server_key_version: "encryptionkey@0"
I20260806 12:28:33.980589 31996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:33.995458 31996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:33.999285 31996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:33.999531 31996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:34.008121 31996 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.125:34105
I20260806 12:28:34.008433 32092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.125:34105 every 8 connection(s)
I20260806 12:28:34.009559 31996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/data/info.pb
I20260806 12:28:34.012272 31988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } attempt: 3
I20260806 12:28:34.013547 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 31996
I20260806 12:28:34.013700 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-1/wal/instance
I20260806 12:28:34.014106 29513 external_mini_cluster.cc:1428] Setting key 5e62ecbedc039c18acc0bcfdf0a9ef44
I20260806 12:28:34.014285 32095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:34.021363 32095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 12:28:34 dist-test-slave-lgr5 krb5kdc[31867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019313, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.210.124@KRBTEST.COM; defaulting to no policy
I20260806 12:28:34.029111 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.013217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:34105 (local address 127.0.0.1:34750)
0806 12:28:34.013533 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.013663 (+ 130us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:34.013795 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:34.018193 (+ 4398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:34.018198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:34.018463 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:34.018674 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.018685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.020191 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.020196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.020959 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.020977 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.021917 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.021921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.022046 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.022656 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:34.022671 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:34.024154 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:34.026719 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.026724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.026727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.027041 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.027504 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.027507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.027510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.027604 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.028589 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:34.028592 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:34.028723 (+ 131us) client_negotiation.cc:769] Sending connection context
0806 12:28:34.028814 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 12:28:34.028965 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.210.124@KRBTEST.COM" created.
I20260806 12:28:34.030818 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.014654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34750 (local address 127.28.210.125:34105)
0806 12:28:34.015434 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.015473 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.015528 (+ 55us) server_negotiation.cc:407] Connection header received
0806 12:28:34.016611 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.016683 (+ 72us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.017661 (+ 978us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.018221 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.018872 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.019990 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.021142 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.021718 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.022059 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.024433 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.024440 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.024457 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.024491 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.026559 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.027191 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.027197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.027203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.027352 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.027701 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.027704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.027706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.028439 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.028585 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.029004 (+ 419us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.029670 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":232,"threads_started":1}
Aug 06 12:28:34 dist-test-slave-lgr5 krb5kdc[31867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019313, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.125@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:28:34.033588 31988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:28:34.034466 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.034122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51604)
0806 12:28:34.034295 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.034357 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 12:28:34.035214 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.034927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51612)
0806 12:28:34.035086 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.035139 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260806 12:28:34.035360 31988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:36067: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
I20260806 12:28:34.036343 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.024123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58132 (local address 127.28.210.126:34209)
0806 12:28:34.024289 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.024294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.024803 (+ 509us) server_negotiation.cc:407] Connection header received
0806 12:28:34.025074 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.025078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.025145 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.025230 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.026223 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.027657 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.028695 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.029035 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.029158 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.031960 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.031965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.031967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.031995 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.033778 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.034420 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.034424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.034426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.034533 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.035110 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.035114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.035116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.035542 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.035627 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.036114 (+ 487us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.036241 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 12:28:34.036516 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.024046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34209 (local address 127.0.0.1:58132)
0806 12:28:34.024636 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.024709 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:34.024913 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:34.025292 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:34.025299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:34.025883 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:34.026071 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.026079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.027808 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.027812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.028565 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.028577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.029423 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.029428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.029540 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.030055 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:34.030067 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:34.031781 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:34.033899 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.033909 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.033915 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.034290 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.034865 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.034869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.034871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.034991 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.035657 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:34.035666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:34.035848 (+ 182us) client_negotiation.cc:769] Sending connection context
0806 12:28:34.036023 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 12:28:34.036297 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":178,"threads_started":1}
I20260806 12:28:34.037809 32095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 12:28:34 dist-test-slave-lgr5 krb5kdc[31867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019313, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.125@KRBTEST.COM for kudu/127.28.210.125@KRBTEST.COM
I20260806 12:28:34.050170 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.038409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:34105 (local address 127.0.0.1:34762)
0806 12:28:34.038601 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.038616 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:34.038717 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:34.038947 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:34.038950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:34.039150 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:34.039293 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.039301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.040073 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.040106 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.040980 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.040997 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.041580 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.041585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.041702 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.042223 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:34.042239 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:34.045828 (+ 3589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:34.048010 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.048014 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.048016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.048354 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.048723 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.048727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.048729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.048886 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.049455 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:34.049458 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:34.049565 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 12:28:34.049670 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 12:28:34.049916 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 12:28:34.050170 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.038450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34762 (local address 127.28.210.125:34105)
0806 12:28:34.038578 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.038582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.038662 (+ 80us) server_negotiation.cc:407] Connection header received
0806 12:28:34.038802 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.038804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.038842 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.038903 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.039397 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.039956 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.041146 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.041436 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.041535 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.046043 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.046047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.046050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.046082 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.047847 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.048465 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.048470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.048472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.048578 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.049002 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.049006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.049008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.049331 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.049445 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.049753 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.050002 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.210.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.124.keytab.
Entry for principal kudu/127.28.210.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.124.keytab.
I20260806 12:28:34.051291 32095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:28:34.052284 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.051866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51616)
0806 12:28:34.052067 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.052201 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 12:28:34.052631 29513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPswvRx/build/debug/bin/kudu
/tmp/dist-test-taskPswvRx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.124:36067
--webserver_interface=127.28.210.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.124.keytab
--principal=kudu/127.28.210.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.210.84:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.210.126:34209,127.28.210.125:34105,127.28.210.124:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 12:28:34.052759 32017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
I20260806 12:28:34.053962 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.053660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51628)
0806 12:28:34.053817 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.053873 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 12:28:34.054258 32095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:36067: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
I20260806 12:28:34.066546 31988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } attempt: 1
I20260806 12:28:34.067543 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.067209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51638)
0806 12:28:34.067387 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.067453 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 12:28:34.068572 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.068238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51646)
0806 12:28:34.068448 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.068502 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260806 12:28:34.068713 31988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:36067: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
I20260806 12:28:34.103524 32095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } attempt: 1
I20260806 12:28:34.104501 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.104192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51648)
0806 12:28:34.104333 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.104419 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 12:28:34.105242 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.105040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51658)
0806 12:28:34.105136 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.105182 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 12:28:34.105389 32095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:36067: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
I20260806 12:28:34.143867 31988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } attempt: 2
I20260806 12:28:34.144894 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.144541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51664)
0806 12:28:34.144736 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.144806 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 12:28:34.145675 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.145445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51680)
0806 12:28:34.145561 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.145610 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 12:28:34.145821 31988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:36067: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
I20260806 12:28:34.173570 32095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } attempt: 2
I20260806 12:28:34.174540 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.174150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51692)
0806 12:28:34.174339 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.174415 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 12:28:34.175446 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.175189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51700)
0806 12:28:34.175307 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.175381 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 12:28:34.175576 32095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:36067: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
W20260806 12:28:34.180009 32107 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 12:28:34.180385 32107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 12:28:34.180416 32107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 12:28:34.180457 32107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 12:28:34.184813 32107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 12:28:34.184916 32107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 12:28:34.184942 32107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 12:28:34.184973 32107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 12:28:34.185003 32107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 12:28:34.190956 32107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.210.84:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.210.126:34209,127.28.210.125:34105,127.28.210.124:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/krb5kdc/kudu_127.28.210.124.keytab
--principal=kudu/127.28.210.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.210.124:36067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.210.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 11:52:03 UTC on ccycloud.quasar-qsdxvn.root.comops.site
build id 4570386
I20260806 12:28:34.192176 32107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 12:28:34.193316 32107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 12:28:34.202919 32117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:34.202939 32116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 12:28:34.202917 32119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 12:28:34.203086 32107 server_base.cc:1034] running on GCE node
I20260806 12:28:34.204006 32107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 12:28:34.205305 32107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 12:28:34.206487 32107 hybrid_clock.cc:648] HybridClock initialized: now 1786019314206460 us; error 51 us; skew 500 ppm
Aug 06 12:28:34 dist-test-slave-lgr5 krb5kdc[31867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 12:28:34.209966 32107 init.cc:377] Logged in from keytab as kudu/127.28.210.124@KRBTEST.COM (short username kudu)
I20260806 12:28:34.211246 32107 webserver.cc:460] Webserver started at http://127.28.210.124:33107/ using document root <none> and password file <none>
I20260806 12:28:34.211977 32107 fs_manager.cc:359] Metadata directory not provided
I20260806 12:28:34.212065 32107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 12:28:34.212380 32107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 12:28:34.215103 32107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/data/instance:
uuid: "474b3afffd0f4ff6880e4b88713bb86a"
format_stamp: "Formatted at 2026-08-06 12:28:34 on dist-test-slave-lgr5"
server_key: "39e6e686c1ab535f8fa98201003679c5"
server_key_iv: "51e7b41a5528b6071be7f4933b18c7d7"
server_key_version: "encryptionkey@0"
I20260806 12:28:34.215878 32107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/wal/instance:
uuid: "474b3afffd0f4ff6880e4b88713bb86a"
format_stamp: "Formatted at 2026-08-06 12:28:34 on dist-test-slave-lgr5"
server_key: "39e6e686c1ab535f8fa98201003679c5"
server_key_iv: "51e7b41a5528b6071be7f4933b18c7d7"
server_key_version: "encryptionkey@0"
I20260806 12:28:34.221113 32107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 12:28:34.224488 32131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 12:28:34.226051 32107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 12:28:34.226210 32107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/data,/tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/wal
uuid: "474b3afffd0f4ff6880e4b88713bb86a"
format_stamp: "Formatted at 2026-08-06 12:28:34 on dist-test-slave-lgr5"
server_key: "39e6e686c1ab535f8fa98201003679c5"
server_key_iv: "51e7b41a5528b6071be7f4933b18c7d7"
server_key_version: "encryptionkey@0"
I20260806 12:28:34.226369 32107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 12:28:34.244982 31988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } attempt: 3
I20260806 12:28:34.245994 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.245613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51712)
0806 12:28:34.245797 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.245875 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 12:28:34.246070 32107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 12:28:34.246917 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.246647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51728)
0806 12:28:34.246783 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.246841 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 12:28:34.247097 31988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.210.124:36067: Network error: Client connection negotiation failed: client connection to 127.28.210.124:36067: connect: Connection refused (error 111)
I20260806 12:28:34.249945 32107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 12:28:34.250128 32107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 12:28:34.257723 32107 rpc_server.cc:307] RPC server started. Bound to: 127.28.210.124:36067
I20260806 12:28:34.258023 32202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.210.124:36067 every 8 connection(s)
I20260806 12:28:34.258850 32107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/data/info.pb
I20260806 12:28:34.261925 29513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPswvRx/build/debug/bin/kudu as pid 32107
I20260806 12:28:34.262070 29513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPswvRx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786018995816160-29513-0/minicluster-data/master-2/wal/instance
I20260806 12:28:34.262349 32203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 12:28:34.262615 29513 external_mini_cluster.cc:1428] Setting key 13ccccaceb817975a583a82b2a1c53ef
I20260806 12:28:34.266767 32095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } attempt: 3
I20260806 12:28:34.269026 32203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 12:28:34 dist-test-slave-lgr5 krb5kdc[31867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019313, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.125@KRBTEST.COM for kudu/127.28.210.124@KRBTEST.COM
Aug 06 12:28:34 dist-test-slave-lgr5 krb5kdc[31867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.124@KRBTEST.COM
Aug 06 12:28:34 dist-test-slave-lgr5 krb5kdc[31867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.124@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:28:34.280930 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.267496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51742)
0806 12:28:34.267854 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.267871 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:34.267967 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:34.268633 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:34.268637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:34.268848 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:34.268997 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.269007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.270145 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.270150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.271028 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.271042 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.272424 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.272428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.272534 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.273383 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:34.273400 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:34.275059 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:34.278373 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.278377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.278378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.278613 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.279097 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.279102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.279104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.279229 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.280202 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:34.280206 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:34.280359 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 12:28:34.280504 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 12:28:34.280693 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258}
I20260806 12:28:34.282347 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.267860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.28.210.124:36067)
0806 12:28:34.268340 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.268349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.268366 (+ 17us) server_negotiation.cc:407] Connection header received
0806 12:28:34.268406 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.268409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.268455 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.268568 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.269186 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.270006 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.271189 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.272229 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.272834 (+ 605us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.275310 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.275317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.275328 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.275361 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.278229 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.278734 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.278742 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.278751 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.278932 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.279336 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.279339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.279341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.280004 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.280213 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.280717 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.281104 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":393,"spinlock_wait_cycles":2816,"thread_start_us":258,"threads_started":1}
I20260806 12:28:34.282347 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.264374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.28.210.124:36067)
0806 12:28:34.264970 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.264989 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.265037 (+ 48us) server_negotiation.cc:407] Connection header received
0806 12:28:34.266024 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.266050 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.266539 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.266962 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.268324 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.269805 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.271236 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.272222 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.272388 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.275927 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.275932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.275934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.275967 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.278213 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.278851 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.278854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.278856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.278946 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.279260 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.279264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.279266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.279999 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.281007 (+ 1008us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.281088 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.281188 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":417,"spinlock_wait_cycles":7168,"thread_start_us":189,"threads_started":1}
I20260806 12:28:34.284821 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.271493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.28.210.126:34209)
0806 12:28:34.271664 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.271668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.272159 (+ 491us) server_negotiation.cc:407] Connection header received
0806 12:28:34.272409 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.272413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.272474 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.272533 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.273613 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.274198 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.275312 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.275680 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.275804 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.279610 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.279614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.279617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.279649 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.281522 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.282082 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.282087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.282089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.282197 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.283507 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.283510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.283512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.283807 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.283941 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.284454 (+ 513us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.284628 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 12:28:34.284848 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.271385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.126:34209 (local address 127.0.0.1:58134)
0806 12:28:34.271975 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.272039 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:34.272284 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:34.272727 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:34.272732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:34.273303 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:34.273469 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.273477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.274347 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.274352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.275113 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.275126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.275837 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.275842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.275972 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.276614 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:34.276628 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:34.279414 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:34.281661 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.281674 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.281682 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.281956 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.282298 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.282302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.282304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.282398 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.283956 (+ 1558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:34.283964 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:34.284186 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 12:28:34.284357 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 12:28:34.284629 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":161,"threads_started":1}
I20260806 12:28:34.286465 32203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:28:34.287781 32095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0: Bootstrap starting.
I20260806 12:28:34.290854 32095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0: Neither blocks nor log segments found. Creating new log.
Aug 06 12:28:34 dist-test-slave-lgr5 krb5kdc[31867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.124@KRBTEST.COM for kudu/127.28.210.125@KRBTEST.COM
I20260806 12:28:34.291976 32095 log.cc:824] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:34.295578 32095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0: No bootstrap required, opened a new log
I20260806 12:28:34.296713 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.287062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.125:34105 (local address 127.0.0.1:34768)
0806 12:28:34.287173 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.287190 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:34.287283 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:34.287559 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:34.287561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:34.287740 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:34.287912 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.287923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.288836 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.288841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.289553 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.289568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.290106 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.290110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.290223 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.290765 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:34.290779 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:34.292542 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:34.294760 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.294764 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.294767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.295102 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.295457 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.295460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.295462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.295568 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.296135 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:34.296138 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:34.296259 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 12:28:34.296362 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 12:28:34.296533 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
Aug 06 12:28:34 dist-test-slave-lgr5 krb5kdc[31867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.126@KRBTEST.COM
I20260806 12:28:34.298115 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.287150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.28.210.125:34105)
0806 12:28:34.287286 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.287290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.287307 (+ 17us) server_negotiation.cc:407] Connection header received
0806 12:28:34.287351 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.287354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.287407 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.287507 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.288060 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.288694 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.289714 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.289980 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.290086 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.292766 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.292770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.292774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.292804 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.294590 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.295223 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.295227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.295229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.295329 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.295685 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.295689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.295691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.296006 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.297631 (+ 1625us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.297756 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.297913 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 12:28:34.299366 32203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } has no permanent_uuid. Determining permanent_uuid...
I20260806 12:28:34.299882 32095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } }
I20260806 12:28:34.300230 32095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1611022b894c71ab6ec11d05dae4d0, State: Initialized, Role: FOLLOWER
I20260806 12:28:34.300663 32095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } }
I20260806 12:28:34.301203 32218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } } }
I20260806 12:28:34.301422 32218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:28:34.302300 32095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:34.303570 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.286303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.28.210.126:34209)
0806 12:28:34.286423 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.286427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.291409 (+ 4982us) server_negotiation.cc:407] Connection header received
0806 12:28:34.291577 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.291581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.291654 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.291761 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.292672 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.293570 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.294795 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.295164 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.295275 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.298774 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.298779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.298782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.298817 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.300927 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.301531 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.301536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.301538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.301655 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.302033 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.302038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.302040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.302355 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.302613 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.303067 (+ 454us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.303214 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Aug 06 12:28:34 dist-test-slave-lgr5 krb5kdc[31867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019314, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.124@KRBTEST.COM for kudu/127.28.210.124@KRBTEST.COM
W20260806 12:28:34.308585 32236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:28:34.308691 32236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:28:34.306852 31996 master_runner.cc:186] Error getting master registration for 127.28.210.126:34209: OK, instance_id { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" instance_seqno: 1786019313764047 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 12:28:34.310909 31996 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 12:28:34.311195 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.300036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51752)
0806 12:28:34.300237 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.300253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:34.301147 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:34.301550 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:34.301553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:34.301742 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:34.301855 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.301862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.302882 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.302887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.303682 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.303696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.304383 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.304388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.304478 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.305112 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:34.305131 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:34.306836 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:34.309331 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.309335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.309338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.309614 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.309960 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.309963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.309965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.310055 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.310621 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:34.310624 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:34.310731 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 12:28:34.310847 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 12:28:34.311014 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 12:28:34.311794 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.300376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.28.210.124:36067)
0806 12:28:34.300495 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.300500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.301235 (+ 735us) server_negotiation.cc:407] Connection header received
0806 12:28:34.301286 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.301290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.301339 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.301411 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.301978 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.302727 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.303854 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.304239 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.307019 (+ 2780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.307168 (+ 149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.307172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.307175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.307195 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.309175 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.309744 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.309747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.309749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.309855 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.310160 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.310163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.310165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.310496 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.311402 (+ 906us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.311492 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.311622 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 12:28:34.315485 32203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a: Bootstrap starting.
Aug 06 12:28:34 dist-test-slave-lgr5 krb5kdc[31867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.210.125@KRBTEST.COM
I20260806 12:28:34.318871 32203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:34.319952 32203 log.cc:824] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:34.320597 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.305656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.28.210.125:34105)
0806 12:28:34.305786 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.305791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.309093 (+ 3302us) server_negotiation.cc:407] Connection header received
0806 12:28:34.309327 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.309331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.309388 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.309444 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.310445 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.311242 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.312519 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.312962 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.313099 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.316453 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.316458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.316461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.316507 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.318264 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.318876 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.318880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.318882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.319060 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.319414 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.319418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.319420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.319714 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.319928 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.320271 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.320415 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 12:28:34.324381 29513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 12:28:34.326714 32203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a: No bootstrap required, opened a new log
I20260806 12:28:34.325773 31948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d46b26dc456c41e1bafe4c18e1212da8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58140
I20260806 12:28:34.331691 32203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } }
I20260806 12:28:34.332127 32203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474b3afffd0f4ff6880e4b88713bb86a, State: Initialized, Role: FOLLOWER
I20260806 12:28:34.332605 32203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } }
I20260806 12:28:34.333330 32239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } } }
I20260806 12:28:34.333575 32239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:28:34.334486 32203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [sys.catalog]: configured and running, proceeding with master startup.
W20260806 12:28:34.341352 32257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:28:34.341468 32257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:28:34.339722 32107 master_runner.cc:186] Error getting master registration for 127.28.210.126:34209: OK, instance_id { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" instance_seqno: 1786019313764047 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 12:28:34.343791 32107 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 12:28:34.411259 31988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } attempt: 4
Aug 06 12:28:34 dist-test-slave-lgr5 krb5kdc[31867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786019313, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.210.126@KRBTEST.COM for kudu/127.28.210.124@KRBTEST.COM
I20260806 12:28:34.421846 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.412007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.210.124:36067 (local address 127.0.0.1:51758)
0806 12:28:34.412289 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 12:28:34.412304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 12:28:34.412455 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 12:28:34.412735 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 12:28:34.412738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 12:28:34.412988 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0806 12:28:34.413162 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.413170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.414226 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.414230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.414786 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 12:28:34.414797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.415582 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.415586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 12:28:34.415687 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.416291 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 12:28:34.416307 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 12:28:34.417983 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 12:28:34.419946 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.419950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.419953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.420295 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.420605 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 12:28:34.420609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 12:28:34.420612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 12:28:34.420721 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 12:28:34.421192 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 12:28:34.421196 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 12:28:34.421337 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 12:28:34.421448 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 12:28:34.421634 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260806 12:28:34.421846 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.412122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.28.210.124:36067)
0806 12:28:34.412290 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.412295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.412359 (+ 64us) server_negotiation.cc:407] Connection header received
0806 12:28:34.412565 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.412568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.412622 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.412692 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.413309 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.414116 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.414918 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.415320 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.415420 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.418173 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.418178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.418180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.418212 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.419805 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.420400 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.420404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.420406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.420501 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.420844 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.420847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.420849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.421079 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.421187 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.421542 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.421670 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 12:28:34.426031 31988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8: Bootstrap starting.
I20260806 12:28:34.431013 31988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8: Neither blocks nor log segments found. Creating new log.
I20260806 12:28:34.432219 31988 log.cc:824] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8: Log is configured to *not* fsync() on all Append() calls
I20260806 12:28:34.434962 31988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8: No bootstrap required, opened a new log
I20260806 12:28:34.438556 31988 raft_consensus.cc:348] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } }
I20260806 12:28:34.438838 31988 raft_consensus.cc:729] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d46b26dc456c41e1bafe4c18e1212da8, State: Initialized, Role: FOLLOWER
I20260806 12:28:34.439203 31988 consensus_queue.cc:260] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } }
I20260806 12:28:34.440949 31988 sys_catalog.cc:564] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 12:28:34.441004 31948 raft_consensus.cc:482] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 12:28:34.441128 31948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:34.442806 31948 raft_consensus.cc:504] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } }
W20260806 12:28:34.448204 32274 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 12:28:34.448335 32274 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 12:28:34.448657 32260 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } } }
I20260806 12:28:34.448796 32260 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:28:34.450163 32058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d46b26dc456c41e1bafe4c18e1212da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1d1611022b894c71ab6ec11d05dae4d0"
I20260806 12:28:34.450438 32058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:34.452005 32058 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d46b26dc456c41e1bafe4c18e1212da8 in term 1.
I20260806 12:28:34.452538 31914 leader_election.cc:302] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d1611022b894c71ab6ec11d05dae4d0, d46b26dc456c41e1bafe4c18e1212da8; no voters:
I20260806 12:28:34.453063 32260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 12:28:34.453490 32260 raft_consensus.cc:686] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [term 1 LEADER]: Becoming Leader. State: Replica: d46b26dc456c41e1bafe4c18e1212da8, State: Running, Role: LEADER
I20260806 12:28:34.453534 31887 master_runner.cc:263] No leader master found from master d46b26dc456c41e1bafe4c18e1212da8
I20260806 12:28:34.453671 31887 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 12:28:34.454052 32260 consensus_queue.cc:237] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } }
I20260806 12:28:34.454322 32164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d46b26dc456c41e1bafe4c18e1212da8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "474b3afffd0f4ff6880e4b88713bb86a"
I20260806 12:28:34.454612 32164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [term 0 FOLLOWER]: Advancing to term 1
I20260806 12:28:34.456359 32164 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d46b26dc456c41e1bafe4c18e1212da8 in term 1.
I20260806 12:28:34.456630 32260 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46b26dc456c41e1bafe4c18e1212da8. Latest consensus state: current_term: 1 leader_uuid: "d46b26dc456c41e1bafe4c18e1212da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } } }
I20260806 12:28:34.456852 32260 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:34.456858 31948 leader_election.cc:288] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [CANDIDATE]: Term 1 election: Requested vote from peers 1d1611022b894c71ab6ec11d05dae4d0 (127.28.210.125:34105), 474b3afffd0f4ff6880e4b88713bb86a (127.28.210.124:36067)
I20260806 12:28:34.459373 32278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 12:28:34.460857 32278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 12:28:34.467708 32164 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [term 1 FOLLOWER]: Refusing update from remote peer d46b26dc456c41e1bafe4c18e1212da8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:28:34.469321 32058 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [term 1 FOLLOWER]: Refusing update from remote peer d46b26dc456c41e1bafe4c18e1212da8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 12:28:34.472424 32276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:28:34.473758 32276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 12:28:34.479293 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.461620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.28.210.124:36067)
0806 12:28:34.461757 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.461762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.468747 (+ 6985us) server_negotiation.cc:407] Connection header received
0806 12:28:34.469254 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.469259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.469383 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.469454 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.470759 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.471459 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.472911 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.473222 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.473302 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.475486 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.475490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.475492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.475526 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.477208 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.477771 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.477776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.477777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.477879 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.478193 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.478196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.478198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.478468 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.478580 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.478979 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.479128 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 12:28:34.479233 32239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46b26dc456c41e1bafe4c18e1212da8. Latest consensus state: current_term: 1 leader_uuid: "d46b26dc456c41e1bafe4c18e1212da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } } }
I20260806 12:28:34.479368 32239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:28:34.487367 32218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46b26dc456c41e1bafe4c18e1212da8. Latest consensus state: current_term: 1 leader_uuid: "d46b26dc456c41e1bafe4c18e1212da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } } }
I20260806 12:28:34.487550 32218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:28:34.491698 32276 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d46b26dc456c41e1bafe4c18e1212da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } } }
I20260806 12:28:34.491901 32276 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:34.492054 32289 mvcc.cc:205] Tried to move back new op lower bound from 7315535112050008064 to 7315535112009351168. Current Snapshot: MvccSnapshot[applied={T|T < 7315535112050008064}]
I20260806 12:28:34.492390 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.460621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.28.210.125:34105)
0806 12:28:34.460765 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.460771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.465382 (+ 4611us) server_negotiation.cc:407] Connection header received
0806 12:28:34.465601 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.465605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.465676 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.465736 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.471347 (+ 5611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.472073 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.482462 (+ 10389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.482775 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.482897 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.484794 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.484799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.484801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.484834 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.489821 (+ 4987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.490308 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.490312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.490314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.490424 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.490738 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.490741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.490742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.490982 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.491105 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.491849 (+ 744us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.492221 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 12:28:34.493738 32239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d46b26dc456c41e1bafe4c18e1212da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } } }
I20260806 12:28:34.493880 32239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:28:34.494366 32290 mvcc.cc:205] Tried to move back new op lower bound from 7315535112050008064 to 7315535112009351168. Current Snapshot: MvccSnapshot[applied={T|T < 7315535112050008064}]
I20260806 12:28:34.494902 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:34.459798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58152 (local address 127.28.210.126:34209)
0806 12:28:34.459922 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.459927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.463483 (+ 3556us) server_negotiation.cc:407] Connection header received
0806 12:28:34.463718 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.463722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.463800 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.463864 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.469904 (+ 6040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.470703 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.484692 (+ 13989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:34.485055 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:34.485171 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:34.486788 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 12:28:34.486792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 12:28:34.486795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 12:28:34.486830 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 12:28:34.488619 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.489150 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.489155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.489157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.489256 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 12:28:34.489578 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 12:28:34.489582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 12:28:34.489584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 12:28:34.489832 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 12:28:34.489962 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:34.494602 (+ 4640us) server_negotiation.cc:299] Negotiation successful
0806 12:28:34.494739 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 12:28:34.496703 32278 catalog_manager.cc:1347] Generated new cluster ID: 04d156491f0442448910edba5057d5be
I20260806 12:28:34.496779 32278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260806 12:28:34.501489 32024 tablet.cc:2369] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 12:28:34.505544 32260 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d46b26dc456c41e1bafe4c18e1212da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } } }
I20260806 12:28:34.505752 32260 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8 [sys.catalog]: This master's current role is: LEADER
I20260806 12:28:34.507465 32295 mvcc.cc:205] Tried to move back new op lower bound from 7315535112050008064 to 7315535112009351168. Current Snapshot: MvccSnapshot[applied={T|T < 7315535112050008064}]
I20260806 12:28:34.507370 32218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d46b26dc456c41e1bafe4c18e1212da8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46b26dc456c41e1bafe4c18e1212da8" member_type: VOTER last_known_addr { host: "127.28.210.126" port: 34209 } } peers { permanent_uuid: "1d1611022b894c71ab6ec11d05dae4d0" member_type: VOTER last_known_addr { host: "127.28.210.125" port: 34105 } } peers { permanent_uuid: "474b3afffd0f4ff6880e4b88713bb86a" member_type: VOTER last_known_addr { host: "127.28.210.124" port: 36067 } } }
I20260806 12:28:34.507704 32218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 12:28:34.520047 32278 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 12:28:34.522655 32278 catalog_manager.cc:1504] Loading token signing keys...
I20260806 12:28:34.555780 32278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d46b26dc456c41e1bafe4c18e1212da8: Generated new TSK 0
I20260806 12:28:34.557013 32278 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 12:28:34.589782 31991 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:34.588148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.28.210.126:34209)
0806 12:28:34.588512 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.588517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.588586 (+ 69us) server_negotiation.cc:407] Connection header received
0806 12:28:34.588820 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.588824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.588884 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.588940 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.589623 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58168: BlockingRecv error: recv got EOF from 127.0.0.1:58168 (error 108)
Metrics: {"server-negotiator.queue_time_us":253}
W20260806 12:28:34.590659 32214 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:34.588629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.28.210.124:36067)
0806 12:28:34.588757 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.588761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.589649 (+ 888us) server_negotiation.cc:407] Connection header received
0806 12:28:34.589814 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.589818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.589877 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.589958 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.590503 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51762: BlockingRecv error: recv got EOF from 127.0.0.1:51762 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 12:28:34.590826 32098 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:34.588878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34796 (local address 127.28.210.125:34105)
0806 12:28:34.589032 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.589037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.590029 (+ 992us) server_negotiation.cc:407] Connection header received
0806 12:28:34.590117 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.590120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.590178 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.590259 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.590702 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34796: BlockingRecv error: recv got EOF from 127.0.0.1:34796 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 12:28:34.594027 32098 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:34.593014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.28.210.125:34105)
0806 12:28:34.593135 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.593139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.593213 (+ 74us) server_negotiation.cc:407] Connection header received
0806 12:28:34.593353 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.593356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.593440 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.593531 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.593912 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34806: BlockingRecv error: recv got EOF from 127.0.0.1:34806 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 12:28:34.598130 32098 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:34.597238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34820 (local address 127.28.210.125:34105)
0806 12:28:34.597367 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.597372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.597386 (+ 14us) server_negotiation.cc:407] Connection header received
0806 12:28:34.597443 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.597447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.597499 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.597585 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.598011 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34820: BlockingRecv error: recv got EOF from 127.0.0.1:34820 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260806 12:28:34.603058 32098 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:34.601892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34834 (local address 127.28.210.125:34105)
0806 12:28:34.602032 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.602037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.602165 (+ 128us) server_negotiation.cc:407] Connection header received
0806 12:28:34.602305 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.602307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.602356 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.602425 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.602919 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34834: BlockingRecv error: recv got EOF from 127.0.0.1:34834 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 12:28:34.611353 32098 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:34.609816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.28.210.125:34105)
0806 12:28:34.610024 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.610028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.610145 (+ 117us) server_negotiation.cc:407] Connection header received
0806 12:28:34.610392 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.610396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.610460 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.610563 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.611207 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34848: BlockingRecv error: recv got EOF from 127.0.0.1:34848 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260806 12:28:34.623683 32098 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:34.622276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.28.210.125:34105)
0806 12:28:34.622445 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.622450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.622607 (+ 157us) server_negotiation.cc:407] Connection header received
0806 12:28:34.622746 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.622749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.622819 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.622957 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.623489 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34850: BlockingRecv error: recv got EOF from 127.0.0.1:34850 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 12:28:34.643782 32098 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:34.642339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.28.210.125:34105)
0806 12:28:34.642538 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.642545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.642727 (+ 182us) server_negotiation.cc:407] Connection header received
0806 12:28:34.642885 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.642887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.642947 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.643076 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.643616 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34854: BlockingRecv error: recv got EOF from 127.0.0.1:34854 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 12:28:34.679800 32098 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:34.678435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34858 (local address 127.28.210.125:34105)
0806 12:28:34.678697 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.678703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.678854 (+ 151us) server_negotiation.cc:407] Connection header received
0806 12:28:34.679025 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.679028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.679082 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.679156 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.679680 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34858: BlockingRecv error: recv got EOF from 127.0.0.1:34858 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
W20260806 12:28:34.747359 32098 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:34.746152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.28.210.125:34105)
0806 12:28:34.746289 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.746302 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.746458 (+ 156us) server_negotiation.cc:407] Connection header received
0806 12:28:34.746613 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.746616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.746667 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.746743 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.747221 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34872: BlockingRecv error: recv got EOF from 127.0.0.1:34872 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 12:28:34.879071 32098 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:34.877796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.28.210.125:34105)
0806 12:28:34.878003 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:34.878008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:34.878130 (+ 122us) server_negotiation.cc:407] Connection header received
0806 12:28:34.878297 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:34.878300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:34.878359 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:34.878433 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:34.878961 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34874: BlockingRecv error: recv got EOF from 127.0.0.1:34874 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260806 12:28:35.139060 32098 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 12:28:35.137747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.28.210.125:34105)
0806 12:28:35.137916 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:35.137920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:35.137957 (+ 37us) server_negotiation.cc:407] Connection header received
0806 12:28:35.138136 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:35.138140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:35.138198 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:35.138274 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 12:28:35.138901 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34876: BlockingRecv error: recv got EOF from 127.0.0.1:34876 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 12:28:35.310346 32236 catalog_manager.cc:1259] Loaded cluster ID: 04d156491f0442448910edba5057d5be
I20260806 12:28:35.310442 32236 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0: loading cluster ID for follower catalog manager: success
I20260806 12:28:35.312448 32236 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0: acquiring CA information for follower catalog manager: success
I20260806 12:28:35.313915 32236 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d1611022b894c71ab6ec11d05dae4d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 12:28:35.343106 32257 catalog_manager.cc:1259] Loaded cluster ID: 04d156491f0442448910edba5057d5be
I20260806 12:28:35.343209 32257 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a: loading cluster ID for follower catalog manager: success
I20260806 12:28:35.345129 32257 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a: acquiring CA information for follower catalog manager: success
I20260806 12:28:35.346438 32257 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 474b3afffd0f4ff6880e4b88713bb86a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 12:28:35.659888 32405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:35.653551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.28.210.125:34105)
0806 12:28:35.653830 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:35.653836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:35.653875 (+ 39us) server_negotiation.cc:407] Connection header received
0806 12:28:35.654088 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:35.654091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:35.654255 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:35.654362 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:28:35.654701 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:35.655294 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:35.656790 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:35.657091 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:35.657450 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:35.658204 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:28:35.659044 (+ 840us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:28:35.659188 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:35.659640 (+ 452us) server_negotiation.cc:299] Negotiation successful
0806 12:28:35.659731 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20260806 12:28:35.667853 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:35.663716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.28.210.124:36067)
0806 12:28:35.664000 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:35.664004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:35.664019 (+ 15us) server_negotiation.cc:407] Connection header received
0806 12:28:35.664172 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:35.664175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:35.664317 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:35.664425 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:28:35.664669 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:35.665212 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:35.666113 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:35.666341 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:35.666421 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:35.666791 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:28:35.667298 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:28:35.667412 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:35.667644 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 12:28:35.667713 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260806 12:28:35.675709 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:35.671624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51772 (local address 127.28.210.124:36067)
0806 12:28:35.671739 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:35.671743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:35.671898 (+ 155us) server_negotiation.cc:407] Connection header received
0806 12:28:35.672125 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:35.672128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:35.672188 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:35.672265 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:28:35.672508 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:35.673236 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:35.674408 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:35.674649 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:35.674747 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:35.675144 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:28:35.675245 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:28:35.675318 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:35.675496 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 12:28:35.675571 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 12:28:35.676661 32405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:35.672031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34906 (local address 127.28.210.125:34105)
0806 12:28:35.672198 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:35.672202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:35.672607 (+ 405us) server_negotiation.cc:407] Connection header received
0806 12:28:35.672706 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:35.672709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:35.672770 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:35.672847 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:28:35.673091 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:35.673634 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:35.675090 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:35.675419 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:35.675511 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:35.675960 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:28:35.676055 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:28:35.676153 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:35.676419 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 12:28:35.676502 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 12:28:35.676699 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 12:28:35.671439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.28.210.126:34209)
0806 12:28:35.671702 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 12:28:35.671707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 12:28:35.671989 (+ 282us) server_negotiation.cc:407] Connection header received
0806 12:28:35.672300 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 12:28:35.672304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 12:28:35.672526 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 12:28:35.672651 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 12:28:35.672983 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:35.673735 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:35.675063 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 12:28:35.675352 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 12:28:35.675466 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 12:28:35.675838 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 12:28:35.676282 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 12:28:35.676385 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 12:28:35.676470 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 12:28:35.676543 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
I20260806 12:28:35.681187 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 31887
I20260806 12:28:35.708504 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 31996
I20260806 12:28:35.729745 29513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPswvRx/build/debug/bin/kudu with pid 32107
2026-08-06T12:28:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3518 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (7015 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (319891 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST